[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 754169881 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002400] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007755] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008029] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009017] pid_max: default: 32768 minimum: 301 [ 0.010168] LSM: Security Framework initializing [ 0.012050] Yama: becoming mindful. [ 0.013047] SELinux: Initializing. [ 0.014130] *** VALIDATE selinux *** [ 0.024165] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029468] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030196] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032148] *** VALIDATE tmpfs *** [ 0.034408] *** VALIDATE proc *** [ 0.035267] *** VALIDATE cgroup *** [ 0.036011] *** VALIDATE cgroup2 *** [ 0.037312] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039130] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042038] Spectre V2 : User space: Vulnerable [ 0.043009] Speculative Store Bypass: Vulnerable [ 0.046688] debug: unmapping init [mem 0xffffffffb1459000-0xffffffffb1460fff] [ 0.049000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049749] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050025] ... version: 2 [ 0.051012] ... bit width: 48 [ 0.052025] ... generic registers: 4 [ 0.053013] ... value mask: 0000ffffffffffff [ 0.054012] ... max period: 00007fffffffffff [ 0.055014] ... fixed-purpose events: 3 [ 0.056010] ... event mask: 000000070000000f [ 0.057327] rcu: Hierarchical SRCU implementation. [ 0.059759] smp: Bringing up secondary CPUs ... [ 0.060566] x86: Booting SMP configuration: [ 0.061043] .... node #0, CPUs: #1 #2 #3 [ 0.088490] smp: Brought up 1 node, 4 CPUs [ 0.090015] smpboot: Max logical packages: 1 [ 0.091015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.182393] node 0 deferred pages initialised in 89ms [ 0.187044] devtmpfs: initialized [ 0.190223] x86/mm: Memory block size: 128MB [ 0.201582] gcov: version magic: 0x41383552 [ 0.207451] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.208154] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.212292] pinctrl core: initialized pinctrl subsystem [ 0.214360] [ 0.215000] ************************************************************* [ 0.218019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.221019] ** ** [ 0.224018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.227019] ** ** [ 0.230020] ** This means that this kernel is built to expose internal ** [ 0.233016] ** IOMMU data structures, which may compromise security on ** [ 0.235017] ** your system. ** [ 0.238017] ** ** [ 0.241020] ** If you see this message and you are not debugging the ** [ 0.244018] ** kernel, report this immediately to your vendor! ** [ 0.247019] ** ** [ 0.250017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.253016] ************************************************************* [ 0.257092] NET: Registered protocol family 16 [ 0.259609] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.263075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.266081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.270124] cpuidle: using governor menu [ 0.273000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.275781] PCI: Using configuration type 1 for base access [ 0.278143] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.286700] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.287024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.291116] cryptd: max_cpu_qlen set to 1000 [ 0.293273] ACPI: Added _OSI(Module Device) [ 0.294018] ACPI: Added _OSI(Processor Device) [ 0.296052] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.298017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.303579] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.314179] ACPI: Interpreter enabled [ 0.316088] ACPI: PM: (supports S0 S3 S4 S5) [ 0.318017] ACPI: Using IOAPIC for interrupt routing [ 0.321196] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.325497] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.341315] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.344093] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.348024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.353101] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.362115] acpiphp: Slot [2] registered [ 0.365729] acpiphp: Slot [5] registered [ 0.368304] acpiphp: Slot [6] registered [ 0.372365] acpiphp: Slot [7] registered [ 0.374415] acpiphp: Slot [8] registered [ 0.377294] acpiphp: Slot [9] registered [ 0.379530] acpiphp: Slot [10] registered [ 0.382573] acpiphp: Slot [3] registered [ 0.385170] acpiphp: Slot [4] registered [ 0.387137] acpiphp: Slot [11] registered [ 0.389207] acpiphp: Slot [12] registered [ 0.392140] acpiphp: Slot [13] registered [ 0.395204] acpiphp: Slot [14] registered [ 0.398219] acpiphp: Slot [15] registered [ 0.401161] acpiphp: Slot [16] registered [ 0.403210] acpiphp: Slot [17] registered [ 0.405486] acpiphp: Slot [18] registered [ 0.409371] acpiphp: Slot [19] registered [ 0.411258] acpiphp: Slot [20] registered [ 0.415235] acpiphp: Slot [21] registered [ 0.417235] acpiphp: Slot [22] registered [ 0.420282] acpiphp: Slot [23] registered [ 0.423189] acpiphp: Slot [24] registered [ 0.426453] acpiphp: Slot [25] registered [ 0.428294] acpiphp: Slot [26] registered [ 0.431204] acpiphp: Slot [27] registered [ 0.434239] acpiphp: Slot [28] registered [ 0.437208] acpiphp: Slot [29] registered [ 0.440146] acpiphp: Slot [30] registered [ 0.442214] acpiphp: Slot [31] registered [ 0.445082] PCI host bridge to bus 0000:00 [ 0.447024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.451313] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.456148] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.459075] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.464031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.467031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.470225] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.473120] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.477612] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.489045] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.518992] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.522021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.525019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.527020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.530701] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.534055] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.537050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.540034] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.547018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.567016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.574020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.588510] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.612022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.637022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.707000] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.715213] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.751019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.764018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.783000] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.783000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.785000] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.785000] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.811022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.818832] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.825019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.830020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.842016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.851000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.856015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.870019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.898019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.910000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.926019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.937019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.955020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.976204] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.977000] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.977000] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.977000] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.977000] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.977418] iommu: Default domain type: Passthrough [ 0.978979] SCSI subsystem initialized [ 1.110284] ACPI: bus type USB registered [ 1.112350] usbcore: registered new interface driver usbfs [ 1.116081] usbcore: registered new interface driver hub [ 1.119377] usbcore: registered new device driver usb [ 1.122036] pps_core: LinuxPPS API ver. 1 registered [ 1.125013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.128067] PTP clock support registered [ 1.132097] EDAC MC: Ver: 3.0.0 [ 1.133079] PCI: Using ACPI for IRQ routing [ 1.134000] NetLabel: Initializing [ 1.135010] NetLabel: domain hash size = 128 [ 1.136008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.139074] NetLabel: unlabeled traffic allowed by default [ 1.141130] vgaarb: loaded [ 1.143614] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.147011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.166671] clocksource: Switched to clocksource kvm-clock [ 1.390180] VFS: Disk quotas dquot_6.6.0 [ 1.412502] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.419116] *** VALIDATE ramfs *** [ 1.421689] *** VALIDATE hugetlbfs *** [ 1.430729] pnp: PnP ACPI init [ 1.484223] pnp: PnP ACPI: found 6 devices [ 1.500242] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.502767] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.504532] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.506387] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.508188] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.510108] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.512347] NET: Registered protocol family 2 [ 1.514387] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.519674] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.523964] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.529610] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.533443] TCP: Hash tables configured (established 65536 bind 65536) [ 1.536747] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.540295] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.544439] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.547230] NET: Registered protocol family 1 [ 1.549814] RPC: Registered named UNIX socket transport module. [ 1.552875] RPC: Registered udp transport module. [ 1.554662] RPC: Registered tcp transport module. [ 1.556750] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.559379] NET: Registered protocol family 44 [ 1.561053] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.563552] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.565967] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.568553] PCI: CLS 0 bytes, default 64 [ 1.570615] Unpacking initramfs... [ 4.530244] debug: unmapping init [mem 0xffff9ff2fcc54000-0xffff9ff2fffbffff] [ 4.534693] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.537432] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.540752] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.663542] Initialise system trusted keyrings [ 5.664974] Key type blacklist registered [ 5.666737] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.675102] zbud: loaded [ 5.677955] *** VALIDATE nfs *** [ 5.679026] *** VALIDATE nfs4 *** [ 5.680318] pstore: using deflate compression [ 5.683745] Platform Keyring initialized [ 5.795525] NET: Registered protocol family 38 [ 5.797406] Key type asymmetric registered [ 5.798837] Asymmetric key parser 'x509' registered [ 5.800576] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.803354] io scheduler mq-deadline registered [ 5.804868] io scheduler kyber registered [ 5.806477] io scheduler bfq registered [ 5.808305] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.811281] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.814258] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.817194] ACPI: Power Button [PWRF] [ 5.824261] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.883169] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.962440] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.972249] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.005128] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.108611] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.142277] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.147267] Non-volatile memory driver v1.3 [ 6.148809] Linux agpgart interface v0.103 [ 6.183311] virtio_blk virtio1: [vda] 149960 512-byte logical blocks (76.8 MB/73.2 MiB) [ 6.186368] vda: detected capacity change from 0 to 76779520 [ 6.204937] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.207814] vdb: detected capacity change from 0 to 1073741824 [ 6.241350] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.244201] vdc: detected capacity change from 0 to 2621440000 [ 6.277184] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.280240] vdd: detected capacity change from 0 to 2621440000 [ 6.294936] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.297975] vde: detected capacity change from 0 to 4294967296 [ 6.312662] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.315575] vdf: detected capacity change from 0 to 4294967296 [ 6.321939] libphy: Fixed MDIO Bus: probed [ 6.326733] usbcore: registered new interface driver usbserial_generic [ 6.328613] usbserial: USB Serial support registered for generic [ 6.330672] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.408988] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.412651] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.416719] mousedev: PS/2 mouse device common for all mice [ 6.421367] rtc_cmos 00:05: RTC can wake from S4 [ 6.427617] rtc_cmos 00:05: registered as rtc0 [ 6.430351] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.435144] intel_pstate: CPU model not supported [ 6.437833] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.442689] hid: raw HID events driver (C) Jiri Kosina [ 6.451050] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.452119] usbcore: registered new interface driver usbhid [ 6.459123] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.465430] usbhid: USB HID core driver [ 6.467206] drop_monitor: Initializing network drop monitor service [ 6.469936] Initializing XFRM netlink socket [ 6.472084] NET: Registered protocol family 10 [ 6.475100] Segment Routing with IPv6 [ 6.476475] NET: Registered protocol family 17 [ 6.478662] mpls_gso: MPLS GSO support [ 6.484096] RAS: Correctable Errors collector initialized. [ 6.486328] AVX version of gcm_enc/dec engaged. [ 6.487985] AES CTR mode by8 optimization enabled [ 6.670169] sched_clock: Marking stable (6670147653, 0)->(8685477978, -2015330325) [ 6.678817] registered taskstats version 1 [ 6.683804] Loading compiled-in X.509 certificates [ 6.688790] zswap: loaded using pool lzo/zbud [ 6.739600] Key type big_key registered [ 6.755601] Key type encrypted registered [ 6.757702] ima: No TPM chip found, activating TPM-bypass! [ 6.760270] ima: Allocated hash algorithm: sha1 [ 6.762478] ima: No architecture policies found [ 6.764960] evm: Initialising EVM extended attributes: [ 6.767522] evm: security.selinux [ 6.768970] evm: security.ima [ 6.770330] evm: security.capability [ 6.772077] evm: HMAC attrs: 0x1 [ 6.775168] rtc_cmos 00:05: setting system clock to 2026-09-08 06:57:56 UTC (1788850676) [ 6.783293] debug: unmapping init [mem 0xffffffffb2403000-0xffffffffb25fffff] [ 6.805817] debug: unmapping init [mem 0xffffffffb1182000-0xffffffffb1458fff] [ 6.823147] Write protecting the kernel read-only data: 28672k [ 6.827224] debug: unmapping init [mem 0xffffffffaf803000-0xffffffffaf9fffff] [ 6.830597] debug: unmapping init [mem 0xffffffffb0114000-0xffffffffb01fffff] [ 6.873713] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.884827] systemd[1]: Detected virtualization kvm. [ 6.887367] systemd[1]: Detected architecture x86-64. [ 6.889816] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.920847] systemd[1]: No hostname configured. [ 6.922942] systemd[1]: Set hostname to . [ 6.925612] random: systemd: uninitialized urandom read (16 bytes read) [ 6.928858] systemd[1]: Initializing machine ID from random generator. [ 7.015986] random: ln: uninitialized urandom read (6 bytes read) [ 7.179160] random: systemd: uninitialized urandom read (16 bytes read) [ 7.182846] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 7.251655] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 7.280297] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.529782] device-mapper: uevent: version 1.0.3 [ 9.532157] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK [ 13.105209] random: fast init done ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 13.169528] virtio_net virtio0 ens2: renamed from eth0 [ 13.640262] scsi host0: ata_piix [ 13.815478] scsi host1: ata_piix [ 13.820329] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 13.828079] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 18.091039] dracut-initqueue[508]: RTNETLINK answers: File exists [ 19.540496] random: crng init done [ 19.575312] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 25.019053] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ 26.373007] hrtimer: interrupt took 21542832 ns [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 28.702135] printk: systemd: 25 output lines suppressed due to ratelimiting [ 29.219568] SELinux: Disabled at runtime. [ 29.292139] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 29.300590] systemd[1]: Detected virtualization kvm. [ 29.303373] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 30.777908] systemd[1]: initrd-switch-root.service: Succeeded. [ 30.785992] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 30.874741] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 30.902606] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 30.908878] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 30.919527] systemd[1]: Starting Journal Service... Starting Journal Service... [ 30.936147] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel...[ 32.557096] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 33.881471] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 35.300580] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 35.373863] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 36.183576] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 36.373340] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit)[ 42.768141] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit)[ 43.733372] NFS: Registering the id_resolver key type [ 43.736668] Key type id_resolver registered [ 43.738302] Key type id_legacy registered [ **] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg350-server login: [ 96.502927] libcfs: loading out-of-tree module taints kernel. [ 96.544239] Key type ._llcrypt registered [ 96.546392] Key type .llcrypt registered [ 96.606664] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 109.279660] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 110.253511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 110.260915] alg: No test for adler32 (adler32-zlib) [ 111.407389] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 111.920266] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 113.592277] Key type lgssc registered [ 114.830119] Lustre: Echo OBD driver; http://www.lustre.org/ [ 127.280536] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 154.155138] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 176.101996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 176.178508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 177.733302] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 177.792521] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 178.085474] Lustre: lustre-MDT0000: new disk, initializing [ 178.222208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 178.235775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 185.620315] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.386385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 207.521703] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 207.562451] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 207.574848] Lustre: Skipped 1 previous similar message [ 207.723071] Lustre: lustre-MDT0001: new disk, initializing [ 207.861570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 207.904846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 207.915180] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 213.719212] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.951527] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 228.526949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 228.900144] Lustre: lustre-OST0000: new disk, initializing [ 228.903813] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 228.913592] Lustre: 8467:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 229.012532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 230.123395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 230.132283] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 230.263257] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 235.161550] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.564647] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 249.799627] Lustre: lustre-OST0001: new disk, initializing [ 249.804566] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 249.814995] Lustre: 9537:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 249.895381] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 255.881642] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 259.137205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 259.155600] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 259.243939] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 267.262419] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 273.073564] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 279.574976] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing check_logdir /tmp/testlogs/ [ 284.760353] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing yml_node [ 288.950333] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 291.409794] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 294.287785] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 295.772474] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 03:02:43 EDT 2026 [ 313.122971] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 314.755547] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 330.723979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 330.728341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 330.741930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 331.235343] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 331.246718] Lustre: Skipped 2 previous similar messages [ 331.258872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 331.267122] Lustre: Skipped 3 previous similar messages [ 336.371095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 336.383974] Lustre: Skipped 3 previous similar messages [ 336.740481] Lustre: server umount lustre-MDT0000 complete [ 340.391908] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851010 with bad export cookie 3456401179124449694 [ 340.392577] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 340.398882] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 341.473850] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 341.488061] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 341.491846] Lustre: Skipped 1 previous similar message [ 341.510433] Lustre: Skipped 1 previous similar message [ 345.508272] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 345.512987] Lustre: Skipped 1 previous similar message [ 346.925206] Lustre: server umount lustre-MDT0001 complete [ 350.952397] Lustre: server umount lustre-OST0000 complete [ 355.035641] Lustre: server umount lustre-OST0001 complete [ 362.288414] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 372.154817] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 411.972427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 412.243552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 412.265068] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 412.314645] Lustre: lustre-MDT0000: new disk, initializing [ 412.387789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 412.416941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 417.533733] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 430.259734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 430.437742] Lustre: 15088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 430.560226] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 430.566192] Lustre: Skipped 1 previous similar message [ 430.632318] Lustre: lustre-MDT0001: new disk, initializing [ 430.695331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 430.730607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 430.745694] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 435.717795] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 445.580790] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 451.645476] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 458.705036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 458.957584] Lustre: lustre-OST0000: new disk, initializing [ 458.963056] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 458.970100] Lustre: 17060:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 459.070411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 460.414702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 460.429784] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 460.561541] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 466.171865] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 477.244678] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 483.091315] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 483.361557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 488.430986] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 488.618287] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 491.492046] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 491.502292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 491.523689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 497.326963] Lustre: server umount lustre-OST0000 complete [ 501.729691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 501.762876] Lustre: Skipped 2 previous similar messages [ 507.599565] Lustre: server umount lustre-MDT0000 complete [ 511.463289] LustreError: 15083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851181 with bad export cookie 3456401179124451227 [ 511.470983] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 511.477082] LustreError: 15083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 511.852403] Lustre: server umount lustre-MDT0001 complete [ 517.162319] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 03:06:25 (1788851185) [ 524.481048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 524.878310] LustreError: 19326:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 524.895129] LustreError: 19326:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 524.968545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 529.911509] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 530.411960] LustreError: 19327:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.500935] LustreError: 19326:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 538.230217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 543.002182] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 551.131438] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 557.341513] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 564.686582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 571.553694] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 579.470641] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 586.521383] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 593.890343] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 593.899419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 593.913725] Lustre: Skipped 1 previous similar message [ 593.941495] Lustre: Skipped 9 previous similar messages [ 599.899520] Lustre: server umount lustre-MDT0000 complete [ 603.859824] LustreError: 19307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851273 with bad export cookie 3456401179124452109 [ 603.868612] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 604.103907] Lustre: server umount lustre-MDT0001 complete [ 617.614616] Lustre: server umount lustre-OST0000 complete [ 638.629361] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 648.647144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 649.254818] LustreError: 23706:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 649.316236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 649.330047] Lustre: Skipped 2 previous similar messages [ 653.590476] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 654.820798] LustreError: 23707:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 663.472545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 669.295394] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 673.353480] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 680.912727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 688.685051] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 697.096597] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 697.243374] Lustre: lustre-OST0001: new disk, initializing [ 697.254967] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 697.268383] Lustre: 26021:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 698.586733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 698.603515] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 698.732166] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 703.779964] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 712.744474] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 716.418503] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 729.569427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 729.581051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 729.588333] Lustre: Skipped 4 previous similar messages [ 729.602634] Lustre: Skipped 5 previous similar messages [ 733.887534] Lustre: server umount lustre-MDT0000 complete [ 734.694912] LustreError: 23702:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 734.722560] LustreError: 23702:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 737.646259] LustreError: 23688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851407 with bad export cookie 3456401179124452956 [ 737.653072] LustreError: 23688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 737.655359] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 738.133407] Lustre: server umount lustre-MDT0001 complete [ 752.464700] Lustre: server umount lustre-OST0000 complete [ 755.682388] Lustre: 3649:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788851409/real 1788851409] req@ffff9ff3651e4a80 x1875745996774912/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788851425 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 757.177456] Lustre: server umount lustre-OST0001 complete [ 772.166855] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 780.880218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 781.344076] LustreError: 29042:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 781.442303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 781.450310] Lustre: Skipped 3 previous similar messages [ 786.490463] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 795.250549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 800.419313] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 803.650606] Lustre: 30190:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 811.825724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 819.463507] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 820.737594] LustreError: 30543:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 820.760722] LustreError: 30543:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 828.734183] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 834.978937] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 841.793421] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 845.004124] Lustre: 32067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 851.937673] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 851.954373] LustreError: Skipped 1 previous similar message [ 851.967158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 851.979196] Lustre: Skipped 1 previous similar message [ 851.984376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 851.989729] Lustre: Skipped 1 previous similar message [ 856.677655] Lustre: server umount lustre-MDT0000 complete [ 860.258964] LustreError: 29024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851529 with bad export cookie 3456401179124454251 [ 860.278745] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 860.733609] Lustre: server umount lustre-MDT0001 complete [ 875.348972] Lustre: server umount lustre-OST0000 complete [ 890.872574] Lustre: server umount lustre-OST0001 complete [ 904.429877] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 03:12:51 (1788851571) [ 939.101278] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 948.947051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 949.546393] LustreError: 34873:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 949.574440] LustreError: 34873:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 954.130082] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 963.478156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 968.655301] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 971.991838] Lustre: 36025:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 979.885155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 987.209145] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 997.419248] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1005.076496] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1012.198963] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1015.642508] Lustre: 37904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1020.387364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1020.394327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1020.404127] Lustre: Skipped 3 previous similar messages [ 1020.409119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1020.414394] Lustre: Skipped 4 previous similar messages [ 1026.622318] Lustre: server umount lustre-MDT0000 complete [ 1030.235330] LustreError: 34855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851699 with bad export cookie 3456401179124455357 [ 1030.237154] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1030.245695] LustreError: 34855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1030.607889] Lustre: server umount lustre-MDT0001 complete [ 1045.004736] Lustre: server umount lustre-OST0000 complete [ 1058.438566] Lustre: server umount lustre-OST0001 complete [ 1066.079160] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 03:15:34 (1788851734) [ 1082.029045] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1091.811671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1092.177880] LustreError: 40045:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1092.192467] LustreError: 40045:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 1092.249906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1092.263808] Lustre: Skipped 7 previous similar messages [ 1096.563616] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1103.658102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1108.542756] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1111.346244] Lustre: 41195:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1118.436620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1125.528621] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1134.450535] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1141.924312] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1149.266752] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1152.830950] Lustre: 43071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1153.921355] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1153.924948] Lustre: Skipped 1 previous similar message [ 1157.616477] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1157.643616] Lustre: Skipped 6 previous similar messages [ 1162.946582] LustreError: 43277:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff9ff35643b600) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 1163.125525] Lustre: server umount lustre-MDT0000 complete [ 1166.520868] LustreError: 40025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851836 with bad export cookie 3456401179124456463 [ 1166.528814] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1166.535043] LustreError: 40025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1172.963226] Lustre: server umount lustre-MDT0001 complete [ 1182.406133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1195.696184] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1202.096880] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1203.078134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1207.378996] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1208.836344] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1208.846788] Lustre: Skipped 1 previous similar message [ 1214.684748] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1216.471842] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1222.806402] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1224.332038] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1236.290851] Lustre: server umount lustre-MDT0000 complete [ 1239.622657] LustreError: 40027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851909 with bad export cookie 3456401179124457562 [ 1239.629941] LustreError: 40027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1239.633829] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1239.968846] Lustre: server umount lustre-MDT0001 complete [ 1254.003996] Lustre: server umount lustre-OST0000 complete [ 1267.805805] Lustre: server umount lustre-OST0001 complete [ 1280.744865] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 03:19:08 (1788851948) [ 1297.942821] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1308.627971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1314.062602] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1327.624090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1335.518137] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1338.806408] Lustre: 48844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1344.310768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1350.043377] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1352.868506] LustreError: 49199:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1352.878290] LustreError: 49199:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 24 previous similar messages [ 1357.889098] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1363.542733] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1370.276520] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1374.103858] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1385.094823] Lustre: server umount lustre-MDT0000 complete [ 1390.073301] LustreError: 49228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852059 with bad export cookie 3456401179124458388 [ 1390.084720] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1390.087857] LustreError: 49228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1390.612445] Lustre: server umount lustre-MDT0001 complete [ 1405.155835] Lustre: server umount lustre-OST0000 complete [ 1419.397978] Lustre: server umount lustre-OST0001 complete [ 1430.046609] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 03:21:37 (1788852097) [ 1432.326165] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1434.101371] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 03:21:42 (1788852102) [ 1442.696888] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 1450.319845] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1470.250077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1470.469815] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1470.603728] Lustre: lustre-MDT0000: new disk, initializing [ 1470.706756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1474.586764] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1483.701335] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1490.811308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1491.032651] Lustre: lustre-OST0000: new disk, initializing [ 1491.037503] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1491.045662] Lustre: Skipped 1 previous similar message [ 1491.051402] Lustre: 54917:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1492.307482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1492.313689] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1492.350473] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1496.754495] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1506.299773] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1509.068822] Lustre: DEBUG MARKER: create test files [ 1548.774203] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1548.774853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1548.787818] Lustre: Skipped 10 previous similar messages [ 1548.788748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1548.807504] Lustre: Skipped 22 previous similar messages [ 1550.403204] Lustre: server umount lustre-OST0000 complete [ 1554.193267] Lustre: server umount lustre-MDT0000 complete [ 1564.295587] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1567.219836] Key type lgssc unregistered [ 1567.557873] LNet: 56545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1567.568563] LNetError: 56545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1567.591451] LNet: Removed LNI 192.168.203.150@tcp [ 1568.629385] Key type .llcrypt unregistered [ 1568.631646] Key type ._llcrypt unregistered [ 1600.555494] Key type ._llcrypt registered [ 1600.557538] Key type .llcrypt registered [ 1600.632837] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1601.504346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1601.531910] alg: No test for adler32 (adler32-zlib) [ 1602.547989] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 1602.745509] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1604.432203] Key type lgssc registered [ 1605.649585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1614.106118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1614.142657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1615.888164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1621.881983] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1632.965084] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1640.816635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1641.023158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1649.032461] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1656.603846] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1660.419100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1660.479832] Lustre: 59276:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 510 < left 544, rollback = 9 [ 1660.487103] Lustre: 59276:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/5, destroy: 0/0/0 [ 1660.493969] Lustre: 59276:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1660.509722] Lustre: 59276:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1660.518926] Lustre: 59276:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1660.529138] Lustre: 59276:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1667.162429] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1672.019887] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1680.867377] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1680.875850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1680.886191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1685.986325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1691.135810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1693.152167] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1693.421176] Lustre: server umount lustre-OST0000 complete [ 1698.469376] Lustre: server umount lustre-MDT0000 complete [ 1709.337358] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1711.678515] Key type lgssc unregistered [ 1712.016117] LNet: 60889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1712.028821] LNetError: 60889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1712.050565] LNet: Removed LNI 192.168.203.150@tcp [ 1712.812468] Key type .llcrypt unregistered [ 1712.815411] Key type ._llcrypt unregistered [ 1740.429235] Key type ._llcrypt registered [ 1740.431852] Key type .llcrypt registered [ 1740.551915] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1741.533679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1741.547209] alg: No test for adler32 (adler32-zlib) [ 1742.540385] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 1742.754581] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1744.456222] Key type lgssc registered [ 1745.690082] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1753.398372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1753.414065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1754.852703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1759.120748] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1767.513994] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1776.425221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1776.736922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1779.856303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1780.022633] Lustre: 63693:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1780.032543] Lustre: 63693:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1780.039204] Lustre: 63693:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1780.050575] Lustre: 63693:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1780.061280] Lustre: 63693:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1780.066638] Lustre: 63693:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1784.468458] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1791.179360] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1796.102707] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1805.803592] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1805.804818] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1805.831911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1810.913934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1811.872755] Lustre: server umount lustre-OST0000 complete [ 1816.142888] Lustre: server umount lustre-MDT0000 complete [ 1827.954642] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1830.502863] Key type lgssc unregistered [ 1830.710884] LNet: 65250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1830.725137] LNetError: 65250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1830.745205] LNet: Removed LNI 192.168.203.150@tcp [ 1831.568508] Key type .llcrypt unregistered [ 1831.572810] Key type ._llcrypt unregistered [ 1847.744869] Key type ._llcrypt registered [ 1847.748208] Key type .llcrypt registered [ 1847.841992] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 1865.309950] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1866.780416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1866.906034] alg: No test for adler32 (adler32-zlib) [ 1867.976637] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 1868.290609] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1870.161359] Key type lgssc registered [ 1872.577238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1923.413755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1923.445846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1925.086461] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1925.184094] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1925.401233] Lustre: lustre-MDT0000: new disk, initializing [ 1925.600355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1925.632330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1931.955445] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1944.393451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1944.587307] Lustre: 68402:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1944.680191] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1944.688863] Lustre: Skipped 1 previous similar message [ 1944.788772] Lustre: lustre-MDT0001: new disk, initializing [ 1944.899841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1944.928309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1944.944549] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1948.892640] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1957.979059] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1962.641164] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1968.919822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1969.198114] Lustre: lustre-OST0000: new disk, initializing [ 1969.227146] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1969.249438] Lustre: 70372:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1969.382490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1975.587629] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1979.532249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1979.549343] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1979.760139] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1986.132507] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1992.110898] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1992.361168] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1997.378672] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1997.674177] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2000.355947] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2000.360859] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2000.372498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2000.381899] LustreError: Skipped 1 previous similar message [ 2000.385391] Lustre: Skipped 1 previous similar message [ 2005.474236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2005.481724] Lustre: Skipped 1 previous similar message [ 2006.492388] Lustre: server umount lustre-OST0000 complete [ 2011.627659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2011.644702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.667534] Lustre: Skipped 1 previous similar message [ 2011.677801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2015.714636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2015.715547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2015.741521] Lustre: Skipped 1 previous similar message [ 2017.227928] Lustre: server umount lustre-MDT0000 complete [ 2021.886465] LustreError: 68394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852691 with bad export cookie 12385719912008785021 [ 2021.900824] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2021.914770] LustreError: 68394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2022.399416] Lustre: server umount lustre-MDT0001 complete [ 2033.355803] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 03:31:41 (1788852701) [ 2044.777626] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2086.202935] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 2095.430483] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2128.329626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2128.549822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2128.593489] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2128.649653] Lustre: lustre-MDT0000: new disk, initializing [ 2128.714138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2128.731226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2132.779854] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2141.952745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2142.067340] Lustre: 76389:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2142.118628] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2142.122223] Lustre: Skipped 1 previous similar message [ 2142.179828] Lustre: lustre-MDT0001: new disk, initializing [ 2142.259129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2142.311611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2142.325573] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2146.728208] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2155.632170] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2160.863603] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2167.754703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2168.139759] Lustre: lustre-OST0000: new disk, initializing [ 2168.148748] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2168.157733] Lustre: 78358:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2168.267104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2175.987448] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2176.141771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2176.161588] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2176.331478] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2187.059794] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2191.273703] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2191.466798] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2195.252827] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2195.450711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2196.964941] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2196.972198] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2196.985693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2202.088927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2202.089508] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2207.205889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2207.218510] Lustre: Skipped 3 previous similar messages [ 2211.297282] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2211.468487] Lustre: server umount lustre-OST0000 complete [ 2217.442908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2221.293624] Lustre: server umount lustre-MDT0000 complete [ 2222.567949] LustreError: 77305:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2222.583581] LustreError: 77305:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2225.389216] LustreError: 76382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852895 with bad export cookie 12385719912008785903 [ 2225.391328] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2225.410291] LustreError: 76382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2225.637110] Lustre: server umount lustre-MDT0001 complete [ 2235.842885] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 03:35:03 (1788852903) [ 2246.667820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2247.246378] LustreError: 80772:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2247.346961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2252.785121] LustreError: 80773:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2253.375650] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2257.901975] LustreError: 80772:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2264.077239] LustreError: 80773:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2264.108294] LustreError: 80773:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2265.270220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2270.545232] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2278.258796] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2284.450711] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2291.757914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2292.487868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2292.500550] Lustre: Skipped 1 previous similar message [ 2298.849761] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2306.060224] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2310.784283] Lustre: 83446:0:(genops.c:1773:obd_export_evict_by_uuid()) MGS: evicting 41fd9715-6a39-40db-a9b9-21458bec02bf at adminstrative request [ 2310.952140] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2310.996744] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2320.354282] Lustre: MGS: Client 41fd9715-6a39-40db-a9b9-21458bec02bf (at 0@lo) reconnecting [ 2320.376966] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0xabe2eaa33ac42361 to 0xabe2eaa33ac42601 [ 2320.427919] Lustre: MGC192.168.203.150@tcp: Connection restored to 0@lo (at 0@lo) [ 2349.360159] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2349.672233] Lustre: lustre-OST0001: new disk, initializing [ 2349.675795] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2349.681054] Lustre: 83843:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2355.374348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2355.389496] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2355.540263] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2357.714875] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2367.443793] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2371.043628] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2371.049951] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2371.053081] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2371.053089] Lustre: Skipped 2 previous similar messages [ 2371.064271] LustreError: Skipped 1 previous similar message [ 2371.074191] Lustre: Skipped 2 previous similar messages [ 2384.352208] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2384.510068] Lustre: server umount lustre-OST0001 complete [ 2386.410512] LustreError: 82596:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.435974] LustreError: 82596:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2387.937353] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2402.273478] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2402.578449] Lustre: server umount lustre-OST0000 complete [ 2406.881497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2406.896196] Lustre: Skipped 2 previous similar messages [ 2406.901717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2406.908095] Lustre: Skipped 13 previous similar messages [ 2412.722479] Lustre: server umount lustre-MDT0000 complete [ 2416.706515] LustreError: 80754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853086 with bad export cookie 12385719912008787457 [ 2416.717093] LustreError: 80754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2416.718175] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2416.987839] Lustre: server umount lustre-MDT0001 complete [ 2425.428700] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 03:38:13 (1788853093) [ 2427.146503] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2428.737093] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 03:38:16 (1788853096) [ 2429.917288] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2431.227802] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 03:38:19 (1788853099) [ 2432.339842] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2433.668117] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 03:38:21 (1788853101) [ 2435.261889] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2436.999686] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 03:38:24 (1788853104) [ 2438.665652] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2444.332203] loop: module loaded [ 2444.344835] loop0: detected capacity change from 0 to 4294967296 [ 2444.410490] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2446.163630] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2449.427821] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2452.885520] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2483.610664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2483.651765] LustreError: 87968:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2483.656366] LustreError: 87968:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2483.660818] LustreError: 87968:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2483.666525] LustreError: 87968:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2502.945912] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 2512.688163] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2550.717882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2551.033633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2551.067349] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2551.144480] Lustre: lustre-MDT0000: new disk, initializing [ 2551.399955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2551.404857] Lustre: Skipped 1 previous similar message [ 2551.431042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2557.065838] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2569.011936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2569.178473] Lustre: 90834:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2569.706836] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2573.854261] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2583.809418] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2590.919772] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2599.629840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2599.802265] Lustre: 92808:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2606.703615] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2606.893075] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2607.133589] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2616.491880] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2621.328799] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2621.581845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2626.800966] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2627.001400] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2632.687835] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2632.712760] LustreError: Skipped 1 previous similar message [ 2632.719374] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2632.725639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2632.742315] Lustre: Skipped 2 previous similar messages [ 2632.760366] Lustre: Skipped 3 previous similar messages [ 2635.583293] Lustre: server umount lustre-OST0000 complete [ 2646.415036] Lustre: server umount lustre-MDT0000 complete [ 2648.035104] LustreError: 92156:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2648.056800] LustreError: 92156:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 2650.482858] LustreError: 90826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853320 with bad export cookie 12385719912008788073 [ 2650.492821] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2650.513375] LustreError: 90826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2650.936423] Lustre: server umount lustre-MDT0001 complete [ 2662.355456] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 03:42:09 (1788853329) [ 2670.239957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2675.235551] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2680.329265] LustreError: 95220:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2680.386688] LustreError: 95220:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2684.127765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2684.523191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2684.533390] Lustre: Skipped 3 previous similar messages [ 2689.989550] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2697.607506] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2703.357510] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2710.954687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2718.096563] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2726.772099] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2732.183200] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2732.370142] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2737.316695] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2737.515216] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2746.080557] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2746.203554] Lustre: lustre-OST0001: new disk, initializing [ 2746.205906] Lustre: Skipped 2 previous similar messages [ 2746.208683] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2746.212460] Lustre: Skipped 3 previous similar messages [ 2746.215972] Lustre: 98593:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2751.664137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2751.678939] Lustre: Skipped 2 previous similar messages [ 2751.686638] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2751.745031] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2752.238263] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2762.959585] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2769.047314] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2769.449615] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2776.812665] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2777.279227] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2801.592869] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2830.384639] Lustre: Failing over lustre-MDT0000 [ 2830.929503] Lustre: server umount lustre-MDT0000 complete [ 2831.706088] LustreError: 95215:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2831.735056] LustreError: 95215:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2832.867099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2832.891063] LustreError: Skipped 2 previous similar messages [ 2832.905050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2832.934818] Lustre: Skipped 2 previous similar messages [ 2850.292432] Lustre: 66623:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788853503/real 1788853503] req@000000005176ea38 x1875747839172864/t0(0) o400->MGC192.168.203.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788853519 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2850.307561] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2855.533743] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2855.536281] LDISKFS-fs (dm-0): recovery complete [ 2855.546679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2861.537421] LustreError: 66620:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000086947494 x1875747839182208/t0(0) o250->MGC192.168.203.150@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2862.033847] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2862.305157] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2862.308390] Lustre: 95215:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2866.824844] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2867.178800] Lustre: 95221:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2867.185242] Lustre: 95221:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 1 previous similar message [ 2867.194978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2867.273859] LustreError: 101442:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2867.594183] LustreError: 101442:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2867.949645] LustreError: 101442:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2867.985081] LustreError: 101442:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 1 previous similar message [ 2868.320236] LustreError: 101442:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2868.333152] LustreError: 101442:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 1 previous similar message [ 2868.341938] Lustre: 101442:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2868.380674] Lustre: 101442:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 7 previous similar messages [ 2869.042834] LustreError: 101442:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2869.061831] LustreError: 101442:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 2 previous similar messages [ 2869.400199] LustreError: 101442:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2869.410803] LustreError: 101442:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 2 previous similar messages [ 2870.414529] Lustre: 101442:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2870.420651] Lustre: 101442:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 5 previous similar messages [ 2871.083199] LustreError: 101442:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2871.101733] LustreError: 101442:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 5 previous similar messages [ 2871.435199] LustreError: 101442:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2871.454523] LustreError: 101442:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 5 previous similar messages [ 2874.711677] Lustre: 101442:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2874.725715] Lustre: 101442:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 12 previous similar messages [ 2875.381175] LustreError: 101442:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2875.394047] LustreError: 101442:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 12 previous similar messages [ 2875.715254] LustreError: 101442:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2875.727171] LustreError: 101442:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 12 previous similar messages [ 2882.754680] Lustre: 101442:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2882.766896] Lustre: 101442:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 24 previous similar messages [ 2883.403435] LustreError: 101442:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2883.420255] LustreError: 101442:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 24 previous similar messages [ 2883.755920] LustreError: 101442:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2883.781746] LustreError: 101442:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 24 previous similar messages [ 2898.769233] Lustre: 101442:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2898.788682] Lustre: 101442:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 49 previous similar messages [ 2899.449629] LustreError: 101442:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2899.462929] LustreError: 101442:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 49 previous similar messages [ 2899.784159] LustreError: 101442:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2899.791759] LustreError: 101442:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 49 previous similar messages [ 2922.057272] Lustre: 101442:0:(ldlm_lib.c:2428:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2922.071354] Lustre: 101442:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2922.082505] Lustre: 101442:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client c15dc68b-b307-461d-99ab-aedc9a0400cf@192.168.203.50@tcp [ 2922.109297] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2922.120121] Lustre: 101442:0:(ldlm_lib.c:1948:abort_req_replay_queue()) @@@ aborted: req@00000000f6055114 x1875747819031296/t0(8589934767) o101->c15dc68b-b307-461d-99ab-aedc9a0400cf@192.168.203.50@tcp:392/0 lens 592/0 e 0 to 0 dl 1788853602 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2922.160264] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2922.182570] Lustre: lustre-MDT0000: Denying connection for new client c15dc68b-b307-461d-99ab-aedc9a0400cf (at 192.168.203.50@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2922.191720] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2922.276434] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2922.285138] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2922.288468] Lustre: Skipped 2 previous similar messages [ 2922.376334] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:77 to 0x280000401:577) [ 2922.389108] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:545) [ 2938.340117] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2938.357723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2938.368309] Lustre: Skipped 3 previous similar messages [ 2941.581382] Lustre: server umount lustre-OST0000 complete [ 2960.352322] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2960.695921] Lustre: server umount lustre-OST0001 complete [ 2971.641606] LustreError: 102527:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000d6f3fd28) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2971.859787] Lustre: server umount lustre-MDT0000 complete [ 2974.694188] LustreError: 95221:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2974.714895] LustreError: 95221:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 54 previous similar messages [ 2976.687789] LustreError: 98591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853646 with bad export cookie 12385719912008839292 [ 2976.689153] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2976.700439] LustreError: 98591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2977.209948] Lustre: server umount lustre-MDT0001 complete [ 2991.722746] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 03:47:37 (1788853657) [ 3001.688711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3001.698403] Lustre: *** cfs_fail_loc=197, val=0*** [ 3017.440706] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3022.560970] LustreError: 103421:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.150@tcp: failed processing log, type 4: rc = -110 [ 3027.711764] Lustre: *** cfs_fail_loc=197, val=0*** [ 3027.715942] Lustre: Skipped 65 previous similar messages [ 3048.224330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3048.230705] Lustre: Skipped 3 previous similar messages [ 3054.858398] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3062.082075] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3065.331197] Lustre: server umount lustre-OST0000 complete [ 3079.361063] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 03:49:06 (1788853746) [ 3107.475976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3114.373939] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3124.403910] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3128.939600] LustreError: 105075:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3128.973365] LustreError: 105075:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3128.985632] LustreError: 105075:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 3129.308244] Lustre: server umount lustre-MDT0000 complete [ 3172.138412] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 3181.804534] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3220.474474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3220.909806] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3220.994424] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3221.135701] Lustre: lustre-MDT0000: new disk, initializing [ 3221.284801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3227.061041] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3239.183321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3239.390491] Lustre: 109251:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3239.650402] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3243.822294] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3252.357174] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3258.429684] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3264.947069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3265.316474] Lustre: 111220:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3273.032561] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3275.434734] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3284.325836] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3290.652441] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3290.972179] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3297.702022] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3297.942911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3301.356023] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3301.358429] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3301.374705] Lustre: Skipped 9 previous similar messages [ 3301.381263] LustreError: Skipped 3 previous similar messages [ 3301.396749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3301.406295] Lustre: Skipped 13 previous similar messages [ 3306.277736] Lustre: server umount lustre-OST0000 complete [ 3316.849253] Lustre: server umount lustre-MDT0000 complete [ 3321.847381] LustreError: 109262:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3321.899742] LustreError: 109262:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 3322.020632] LustreError: 109243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853991 with bad export cookie 12385719912008849834 [ 3322.025232] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3322.033640] LustreError: 109243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3322.812585] Lustre: server umount lustre-MDT0001 complete [ 3337.322961] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 03:53:24 (1788854004) [ 3347.146376] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 3350.702798] Key type lgssc unregistered [ 3351.131916] LNet: 113712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3351.137308] LNetError: 113712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3352.182699] LNet: Removed LNI 192.168.203.150@tcp [ 3353.328306] Key type .llcrypt unregistered [ 3353.330822] Key type ._llcrypt unregistered [ 3374.275582] Key type ._llcrypt registered [ 3374.278660] Key type .llcrypt registered [ 3374.426078] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 3393.876729] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3394.695554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3394.714313] alg: No test for adler32 (adler32-zlib) [ 3396.167610] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 3396.499894] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 3398.336399] Key type lgssc registered [ 3399.730389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3446.560492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3446.602489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3447.877770] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3447.952563] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3448.060777] Lustre: lustre-MDT0000: new disk, initializing [ 3448.234610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3448.251754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3453.364083] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3464.753458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3464.932976] Lustre: lustre-OST0000: new disk, initializing [ 3464.935974] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3464.939364] Lustre: Skipped 1 previous similar message [ 3464.943674] Lustre: 117885:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3465.013505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3470.437452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3470.446925] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3470.478911] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3470.863757] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3481.267707] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3481.502319] Lustre: lustre-OST0001: new disk, initializing [ 3481.508147] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3481.515596] Lustre: 118750:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3481.588330] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3487.750647] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3491.081500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3491.097386] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3491.154553] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3501.549436] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3501.564185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3501.587997] Lustre: Skipped 1 previous similar message [ 3501.620416] Lustre: Skipped 1 previous similar message [ 3503.985132] Lustre: server umount lustre-MDT0000 complete [ 3512.572876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3518.144070] LustreError: 117073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854187 with bad export cookie 8562367052488784778 [ 3518.151124] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3518.155376] LustreError: 117073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3518.211466] Lustre: server umount lustre-OST0000 complete [ 3521.816630] Lustre: server umount lustre-OST0001 complete [ 3531.394651] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 03:56:39 (1788854199) [ 3559.610719] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 03:57:07 (1788854227) [ 3572.409346] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3598.024154] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 03:57:45 (1788854265) [ 3633.619177] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 3642.703579] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3687.865860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3688.066368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3688.094645] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3688.155583] Lustre: lustre-MDT0000: new disk, initializing [ 3688.230716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3688.263246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3694.731576] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3708.767141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3709.010425] Lustre: 126505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3709.127785] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3709.132729] Lustre: Skipped 1 previous similar message [ 3709.304567] Lustre: lustre-MDT0001: new disk, initializing [ 3709.476735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3709.539812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3709.571015] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3714.779380] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3725.349859] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3731.311820] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3738.614894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3738.918529] Lustre: lustre-OST0000: new disk, initializing [ 3738.922303] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3738.928604] Lustre: 128474:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3739.011922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3745.522616] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3749.055919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3749.076199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3749.383924] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3760.027053] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3769.024059] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3769.374427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3775.328179] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3775.516432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3780.074280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3780.075051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3780.076410] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3780.087761] Lustre: Skipped 1 previous similar message [ 3785.189397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3785.199890] Lustre: Skipped 1 previous similar message [ 3790.321385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3790.341694] Lustre: Skipped 2 previous similar messages [ 3792.353173] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3792.706379] Lustre: server umount lustre-OST0000 complete [ 3800.553256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3800.558334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3800.582180] Lustre: Skipped 1 previous similar message [ 3804.309964] Lustre: server umount lustre-MDT0000 complete [ 3810.788849] LustreError: 126519:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3810.840695] LustreError: 126519:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3811.275897] LustreError: 126500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854481 with bad export cookie 8562367052488786010 [ 3811.285480] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3811.299789] LustreError: 126500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3811.929337] Lustre: server umount lustre-MDT0001 complete [ 3825.820661] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 04:01:33 (1788854493) [ 3834.155623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3834.590032] LustreError: 130902:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3834.693052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3839.439989] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3839.982538] LustreError: 130903:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3845.090396] LustreError: 130902:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3848.863919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3855.069358] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3867.375916] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3872.994530] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3881.669613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3882.067897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3882.073429] Lustre: Skipped 1 previous similar message [ 3888.827220] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3898.333853] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3914.548475] Lustre: *** cfs_fail_loc=159, val=0*** [ 3914.554452] Lustre: Skipped 1 previous similar message [ 3929.980260] Lustre: lustre-MDT0000: Client 5c1492e0-a3f7-43cb-aa3a-8c66fc0a13a3 (at 192.168.203.50@tcp) reconnecting [ 3931.755345] Lustre: *** cfs_fail_loc=159, val=0*** [ 3931.761761] Lustre: Skipped 4 previous similar messages [ 3948.476679] Lustre: lustre-MDT0000: Client 5c1492e0-a3f7-43cb-aa3a-8c66fc0a13a3 (at 192.168.203.50@tcp) reconnecting [ 3965.410921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3965.418444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3965.440199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3965.453728] Lustre: Skipped 1 previous similar message [ 3966.447929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3971.578745] Lustre: server umount lustre-OST0000 complete [ 3976.676163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3976.684993] Lustre: Skipped 2 previous similar messages [ 3981.800673] LustreError: 133461:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3981.939433] Lustre: server umount lustre-MDT0000 complete [ 3986.639445] LustreError: 130884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854656 with bad export cookie 8562367052488786892 [ 3986.644982] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3986.660587] LustreError: 130884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3987.232793] Lustre: server umount lustre-MDT0001 complete [ 4001.064428] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 4004.724373] Key type lgssc unregistered [ 4005.141423] LNet: 134698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4005.158731] LNetError: 134698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4005.183807] LNet: Removed LNI 192.168.203.150@tcp [ 4006.551093] Key type .llcrypt unregistered [ 4006.554095] Key type ._llcrypt unregistered [ 4034.045557] Key type ._llcrypt registered [ 4034.052257] Key type .llcrypt registered [ 4034.159033] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4035.266530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4035.424127] alg: No test for adler32 (adler32-zlib) [ 4036.497911] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 4036.816565] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 4038.577366] Key type lgssc registered [ 4039.818173] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4048.775734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4048.800291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4050.276606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4054.598217] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4059.623529] LustreError: 136127:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4059.647269] LustreError: 136127:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4064.340331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4064.816607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4069.809917] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4078.133410] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4083.751786] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4091.054392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4091.412252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4094.526443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4098.145189] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4105.902435] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4119.189768] Lustre: *** cfs_fail_loc=159, val=0*** [ 4134.802501] Lustre: lustre-MDT0000: Client 363c9ca5-caec-4204-8e67-9ce179b31488 (at 192.168.203.50@tcp) reconnecting [ 4138.135059] Lustre: *** cfs_fail_loc=159, val=0*** [ 4153.719029] Lustre: lustre-MDT0001: Client 363c9ca5-caec-4204-8e67-9ce179b31488 (at 192.168.203.50@tcp) reconnecting [ 4155.280293] Lustre: *** cfs_fail_loc=159, val=0*** [ 4155.282050] Lustre: Skipped 3 previous similar messages [ 4171.623841] Lustre: lustre-MDT0001: Client 363c9ca5-caec-4204-8e67-9ce179b31488 (at 192.168.203.50@tcp) reconnecting [ 4176.590963] Lustre: 139167:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4179.999788] Lustre: *** cfs_fail_loc=159, val=0*** [ 4180.005340] Lustre: Skipped 5 previous similar messages [ 4195.741668] Lustre: lustre-MDT0000: Client a3870b58-b0d9-47b5-8fb7-cfbccbed31e3 (at 192.168.203.50@tcp) reconnecting [ 4197.650629] Lustre: *** cfs_fail_loc=159, val=0*** [ 4197.656910] Lustre: Skipped 13 previous similar messages [ 4214.142614] Lustre: lustre-MDT0000: Client a3870b58-b0d9-47b5-8fb7-cfbccbed31e3 (at 192.168.203.50@tcp) reconnecting [ 4216.362403] Lustre: 139463:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4232.681767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4232.697908] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4232.702913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4232.710911] Lustre: Skipped 1 previous similar message [ 4232.737946] LustreError: Skipped 1 previous similar message [ 4233.694904] Lustre: server umount lustre-OST0000 complete [ 4237.798777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4237.811693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4237.816864] Lustre: Skipped 1 previous similar message [ 4242.924870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4242.941685] Lustre: Skipped 1 previous similar message [ 4243.737302] Lustre: server umount lustre-MDT0000 complete [ 4247.878806] LustreError: 139412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854917 with bad export cookie 16893927406554056585 [ 4247.879699] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4247.908118] LustreError: 139412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4248.265624] Lustre: server umount lustre-MDT0001 complete [ 4260.690107] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 4263.443485] Key type lgssc unregistered [ 4263.820498] LNet: 140542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4263.829082] LNetError: 140542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4263.841675] LNet: Removed LNI 192.168.203.150@tcp [ 4264.605504] Key type .llcrypt unregistered [ 4264.614272] Key type ._llcrypt unregistered [ 4287.027852] Key type ._llcrypt registered [ 4287.029738] Key type .llcrypt registered [ 4287.210754] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4288.268852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4288.397742] alg: No test for adler32 (adler32-zlib) [ 4289.401245] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 4289.679676] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 4291.384313] Key type lgssc registered [ 4292.789850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4301.183886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4301.206789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4302.754780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4306.875035] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4313.062517] LustreError: 141952:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4313.075746] LustreError: 141952:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4316.041718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4316.608707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4322.246891] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4330.172699] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4336.379499] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4343.903815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4344.306985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4347.402439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4351.124706] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4359.695588] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4361.749432] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4366.571838] Lustre: 144537:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4369.884649] Lustre: 144587:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4371.585287] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4372.729760] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4376.867527] Lustre: 144734:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4377.847429] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4378.726398] Lustre: 144832:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4387.300446] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4387.309501] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4387.333900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4387.348550] Lustre: Skipped 1 previous similar message [ 4389.603594] Lustre: server umount lustre-OST0000 complete [ 4397.546035] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4397.555052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4397.571810] Lustre: Skipped 2 previous similar messages [ 4397.593021] Lustre: Skipped 1 previous similar message [ 4400.683963] Lustre: server umount lustre-MDT0000 complete [ 4402.664262] LustreError: 141953:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4402.695938] LustreError: 141953:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4405.869356] LustreError: 141933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855075 with bad export cookie 3719077918410127458 [ 4405.885844] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4405.899326] LustreError: 141933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4406.698238] Lustre: server umount lustre-MDT0001 complete [ 4423.960501] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 4427.114273] Key type lgssc unregistered [ 4427.344543] LNet: 145913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4427.362651] LNetError: 145913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4427.388427] LNet: Removed LNI 192.168.203.150@tcp [ 4428.289362] Key type .llcrypt unregistered [ 4428.291244] Key type ._llcrypt unregistered [ 4455.443495] Key type ._llcrypt registered [ 4455.447841] Key type .llcrypt registered [ 4455.630611] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4457.242946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4457.482676] alg: No test for adler32 (adler32-zlib) [ 4458.941728] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 4459.387977] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 4461.146387] Key type lgssc registered [ 4463.131747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4473.276123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4473.308469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4475.212246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4480.498531] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4484.582442] LustreError: 147323:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4484.604027] LustreError: 147323:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4489.705232] LustreError: 147324:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4489.992763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4490.506354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4495.705914] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4503.535854] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4510.422777] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4517.043802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4517.435895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4519.494432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4523.382684] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4530.035888] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4535.857551] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4539.790402] Lustre: *** cfs_fail_loc=159, val=0*** [ 4556.113947] Lustre: lustre-MDT0000: Client 7d0a46b4-718f-409e-98cf-d65d094e13ef (at 192.168.203.50@tcp) reconnecting [ 4561.898332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4561.901386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4561.904146] LustreError: Skipped 2 previous similar messages [ 4561.926119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4566.497781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4566.513923] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4566.519177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4566.540220] LustreError: Skipped 5 previous similar messages [ 4567.807184] Lustre: server umount lustre-OST0000 complete [ 4572.645846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4572.652247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4572.661955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4572.664845] Lustre: Skipped 1 previous similar message [ 4576.739166] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4576.739654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4576.787485] Lustre: Skipped 1 previous similar message [ 4578.458914] Lustre: server umount lustre-MDT0000 complete [ 4581.858591] LustreError: 147320:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4581.885195] LustreError: 147320:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4583.864105] LustreError: 147305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855253 with bad export cookie 7525925148955418437 [ 4583.876173] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4583.895404] LustreError: 147305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4585.168532] Lustre: server umount lustre-MDT0001 complete [ 4601.485559] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 4604.626228] Key type lgssc unregistered [ 4605.008771] LNet: 151003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4605.017744] LNetError: 151003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4605.066073] LNet: Removed LNI 192.168.203.150@tcp [ 4606.250107] Key type .llcrypt unregistered [ 4606.256141] Key type ._llcrypt unregistered [ 4633.099940] Key type ._llcrypt registered [ 4633.102330] Key type .llcrypt registered [ 4633.236650] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4634.358660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4634.459057] alg: No test for adler32 (adler32-zlib) [ 4635.571930] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 4635.944992] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 4637.641605] Key type lgssc registered [ 4638.823414] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4646.647867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4646.670762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4648.274498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4653.082787] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4657.641322] LustreError: 152416:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4657.677838] LustreError: 152416:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4661.520061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4662.159708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4666.765760] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4675.266182] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4681.775275] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4689.455612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4690.274215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4692.340912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4697.438266] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4706.694562] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4706.893947] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4727.607714] Lustre: lustre-OST0000: evicting dd62ee88-a1a8-44bd-a4be-87d5e81f1373 (at 192.168.203.50@tcp) by administrative request [ 4727.621312] Lustre: Skipped 1 previous similar message [ 4757.987102] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4758.011644] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4758.013319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4758.055203] Lustre: Skipped 1 previous similar message [ 4762.377853] Lustre: server umount lustre-OST0000 complete [ 4768.231747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4768.237329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4768.260266] Lustre: Skipped 2 previous similar messages [ 4772.718602] Lustre: server umount lustre-MDT0000 complete [ 4773.347614] LustreError: 152415:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4773.375822] LustreError: 152415:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4777.449115] LustreError: 154367:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4777.502735] LustreError: 152395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855447 with bad export cookie 3615006833898170853 [ 4777.508966] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4777.520065] LustreError: 152395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4778.078831] Lustre: server umount lustre-MDT0001 complete [ 4790.571672] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 4793.983054] Key type lgssc unregistered [ 4794.402494] LNet: 156484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4794.408950] LNetError: 156484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4795.431526] LNet: Removed LNI 192.168.203.150@tcp [ 4796.768981] Key type .llcrypt unregistered [ 4796.771149] Key type ._llcrypt unregistered [ 4872.331354] Key type ._llcrypt registered [ 4872.337466] Key type .llcrypt registered [ 4872.440680] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4873.546351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4873.575607] alg: No test for adler32 (adler32-zlib) [ 4874.611595] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 4874.851052] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 4876.549273] Key type lgssc registered [ 4877.937913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4886.001074] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4886.033614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4887.749704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4892.992870] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4898.281729] LustreError: 158972:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4898.315040] LustreError: 158972:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4902.843107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4903.498350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4907.801260] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4915.856315] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4922.415699] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4930.793690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4931.243355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4933.301430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4937.990723] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4944.749173] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4948.672402] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4959.206084] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4959.216732] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4959.228030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4959.229077] LustreError: Skipped 1 previous similar message [ 4959.296346] Lustre: Skipped 1 previous similar message [ 4963.717281] Lustre: server umount lustre-OST0000 complete [ 4969.441322] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4969.446147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4969.461411] Lustre: Skipped 1 previous similar message [ 4969.474348] Lustre: Skipped 2 previous similar messages [ 4974.600341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4974.611220] Lustre: Skipped 1 previous similar message [ 4979.685819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4979.696942] Lustre: Skipped 1 previous similar message [ 4983.264167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4983.520916] Lustre: server umount lustre-MDT0000 complete [ 4985.826803] LustreError: 159993:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4985.848062] LustreError: 159993:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4986.969522] LustreError: 158953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855656 with bad export cookie 18137920336578538945 [ 4986.982595] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4986.985981] LustreError: 158953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4987.390371] Lustre: server umount lustre-MDT0001 complete [ 4999.407914] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5002.379362] Key type lgssc unregistered [ 5002.830603] LNet: 162536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5002.842671] LNetError: 162536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5002.858228] LNet: Removed LNI 192.168.203.150@tcp [ 5003.575283] Key type .llcrypt unregistered [ 5003.578664] Key type ._llcrypt unregistered [ 5037.488784] Key type ._llcrypt registered [ 5037.491081] Key type .llcrypt registered [ 5037.605376] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5038.897402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5038.946651] alg: No test for adler32 (adler32-zlib) [ 5039.939513] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 5040.141654] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 5041.864280] Key type lgssc registered [ 5043.216705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5059.496726] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 04:22:07 (1788855727) [ 5076.470051] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 5086.830648] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5128.586397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5128.622741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5129.958495] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5129.999351] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5130.182311] Lustre: lustre-MDT0000: new disk, initializing [ 5130.317882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5130.386271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5136.338204] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5148.514230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5148.692949] Lustre: 166797:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5148.743413] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5148.748683] Lustre: Skipped 1 previous similar message [ 5148.899378] Lustre: lustre-MDT0001: new disk, initializing [ 5149.029465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5149.087224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5149.099845] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5154.215385] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5163.773498] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5170.410815] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5176.898942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5177.318712] Lustre: lustre-OST0000: new disk, initializing [ 5177.325127] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5177.336668] Lustre: 168767:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5177.447316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5183.756195] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5186.150856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5186.164390] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5186.273964] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5193.925840] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5207.013200] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5207.019513] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5207.040718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5207.075473] Lustre: Skipped 1 previous similar message [ 5212.131277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5212.135676] Lustre: Skipped 2 previous similar messages [ 5212.713907] Lustre: server umount lustre-OST0000 complete [ 5217.250330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5217.260614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5217.269967] Lustre: Skipped 1 previous similar message [ 5217.291938] Lustre: Skipped 1 previous similar message [ 5222.369934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5222.373723] Lustre: Skipped 1 previous similar message [ 5222.734797] Lustre: server umount lustre-MDT0000 complete [ 5226.434877] LustreError: 166792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855896 with bad export cookie 9745608234361672475 [ 5226.450942] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5226.881774] Lustre: server umount lustre-MDT0001 complete [ 5236.045160] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5238.723277] Key type lgssc unregistered [ 5239.070606] LNet: 170675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5239.075498] LNetError: 170675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5239.105269] LNet: Removed LNI 192.168.203.150@tcp [ 5239.864795] Key type .llcrypt unregistered [ 5239.866905] Key type ._llcrypt unregistered [ 5264.662932] Key type ._llcrypt registered [ 5264.664467] Key type .llcrypt registered [ 5264.764664] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5265.363677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5265.397883] alg: No test for adler32 (adler32-zlib) [ 5266.432433] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 5266.574169] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 5268.192336] Key type lgssc registered [ 5269.231552] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5276.521983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5276.557239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5278.098366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5283.932592] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5287.395931] LustreError: 172086:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5287.409328] LustreError: 172086:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5292.526457] LustreError: 172087:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5294.145489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5294.555667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5299.198670] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5307.507644] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5314.000101] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5321.288239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5321.672559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5322.700844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5328.264762] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5336.005984] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5346.596213] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5346.639665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5346.660826] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5346.679619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.693516] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5347.943801] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5347.954693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5347.962516] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5347.971526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.005795] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5348.946910] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5348.959738] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5348.976861] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5348.982346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.767161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.776451] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5349.786468] Lustre: Skipped 1 previous similar message [ 5350.834851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.744755] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5351.754559] Lustre: Skipped 2 previous similar messages [ 5351.775368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5351.787834] Lustre: Skipped 2 previous similar messages [ 5351.806629] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5351.818288] Lustre: Skipped 2 previous similar messages [ 5351.827171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.846674] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5351.863543] Lustre: Skipped 1 previous similar message [ 5353.120462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.346920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.463799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.687813] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5356.691839] Lustre: Skipped 3 previous similar messages [ 5356.721904] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5356.732875] Lustre: Skipped 3 previous similar messages [ 5356.742925] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5356.747384] Lustre: Skipped 3 previous similar messages [ 5356.756402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.769045] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5356.776293] Lustre: Skipped 3 previous similar messages [ 5357.716725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.743346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.624738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.517237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.376519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.323860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.334922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.395392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.433693] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5365.446981] Lustre: Skipped 8 previous similar messages [ 5365.476795] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5365.486958] Lustre: Skipped 8 previous similar messages [ 5365.489543] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5365.500844] Lustre: Skipped 8 previous similar messages [ 5365.504911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.511913] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5365.516590] Lustre: Skipped 8 previous similar messages [ 5366.281891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.270674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.079553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.982233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.880270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.391228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.835672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.229301] LustreError: 172106:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5374.254826] LustreError: 172106:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5374.292502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.725654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.428496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.683277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.321642] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5382.338965] Lustre: Skipped 11 previous similar messages [ 5382.413539] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5382.450994] Lustre: Skipped 11 previous similar messages [ 5382.462936] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5382.470597] Lustre: Skipped 11 previous similar messages [ 5382.479957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.516103] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5382.544870] Lustre: Skipped 11 previous similar messages [ 5384.082103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.905903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.765485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.370114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.465161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.898479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.367527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.650211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.527229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.844772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.309133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.990478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.397910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.231630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.580175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.822908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.908300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.159675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.801120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.712432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.764968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.882492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.009607] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5415.018519] Lustre: Skipped 22 previous similar messages [ 5415.035362] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5415.050655] Lustre: Skipped 22 previous similar messages [ 5415.062113] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5415.072642] Lustre: Skipped 22 previous similar messages [ 5415.079300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.100146] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5415.112655] Lustre: Skipped 22 previous similar messages [ 5416.007125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.984293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.065743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.197044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.196233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.389674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.537034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.748768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.838244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.970699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.060039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.048714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.078845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.130423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.047232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.057049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.194025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.269743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.130511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.094808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.891846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.718742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.673067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.651694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.437396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.352944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.144589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.102348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.915240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.734812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.837178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.738808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.728193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.705990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.746122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.606571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.689785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.758656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.690475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.580616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.477069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.263630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.246873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.224067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.220833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.999970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.006746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.953488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.206850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.245957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.169658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.997065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.016973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.935913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.912398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.846783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.641225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.351351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.156337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.165315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.933949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.828734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.687116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.827625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.681583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.712880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.680781] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5479.686960] Lustre: Skipped 66 previous similar messages [ 5479.709553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5479.723736] Lustre: Skipped 66 previous similar messages [ 5479.726639] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5479.730533] Lustre: Skipped 66 previous similar messages [ 5479.733059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.744344] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5479.755616] Lustre: Skipped 66 previous similar messages [ 5480.775408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.719982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.800527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.919775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.822970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.798174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.644488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.576687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.591868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.804438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.680353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.554524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.544831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.506844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.295093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.526237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.471857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.366558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.244901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.350993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.170930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.083803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.877552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.721877] LustreError: 172106:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5502.731402] LustreError: 172106:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5502.740733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.527905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.366850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.021440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.701485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.464066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.308903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.020920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.794739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.597074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.562460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.321606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.594139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.640575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.533893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.268464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.089269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.862100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.043743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.042720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.216220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.460610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.661124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.711479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.694755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.529095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.279096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.054504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.846702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.807058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.889586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.716975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.717910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.012022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.163274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.030931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.029312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.882387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.160444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.188962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.271625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.257421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.034726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.164719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.149936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.041107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.883695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.573288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.365697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.247217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.098049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.811988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.639524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.277756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.245313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.118488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.580778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.417772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.832353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.100481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.839440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.079200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.076118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.765342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.357984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.260620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.114282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.763521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.562699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.466153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.770937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.811324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.003464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.126272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.457562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.473622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.410198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.259218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.403146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.399722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.350983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.547954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.559291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.617583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.649727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.485795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.221935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.137241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.242362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.448538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.381794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.418483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.421460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.419218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.349272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.408384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.387059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.420658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.384716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.338607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.257698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.222556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.042757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.033369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.960709] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5607.966092] Lustre: Skipped 127 previous similar messages [ 5607.983751] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5607.994967] Lustre: Skipped 127 previous similar messages [ 5608.002573] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5608.006955] Lustre: Skipped 127 previous similar messages [ 5608.012239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.022215] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5608.027405] Lustre: Skipped 127 previous similar messages [ 5609.262338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.409798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.477128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.635224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.651235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.764294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.903058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.071699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.005502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.931544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.912562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.886757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.766696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.772606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.469471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.405666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.212212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.087487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.956589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.913344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.851763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.809335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.725126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.689625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.499778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.434611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.476196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.429774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.384882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.255721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.104958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.037671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.773133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.645733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.574566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.491494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.479652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.605491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.483087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.384437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.333098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.271741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.006469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.810684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.686396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.575641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.374518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.191832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.968156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.780604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.689496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.474476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.429840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.390748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.319126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.323744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.067045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.828994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.514300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.537878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.588345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.503860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.486187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.295476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.233676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.072777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.251217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.227976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.302058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.300163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.303774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.116700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.130548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.981214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.024657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.026773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.864363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.816161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.760425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.839732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.811022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.897308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.927068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.817575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.759966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.839505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.928520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.955784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.243962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.392111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.275479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.370445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.312488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.150932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.917540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.792438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.576542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.436736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.229407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.003091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.974383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.917035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.099715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.752061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.858768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.215404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.585518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.482808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.858586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.018303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.820321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.009977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.941336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.992604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.952847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.858918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.730627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.650640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.505316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.522572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.538662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.857373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.923529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.153448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.139848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.496125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.867170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.236214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.744714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.088619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.173270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.498307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.638809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.229048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.276697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.004158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.604359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.951088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.729460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.108855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.090101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.616170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.807472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.798398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.944250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.947280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.354482] LustreError: 172106:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5761.379603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.385420] LustreError: 172106:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5762.491044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.185714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.541668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.851282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.157886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.955128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.460321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.505945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.865557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.824930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.675186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.744354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.811795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.830940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.971608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.869908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.802777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.524636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.763760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.378477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.995442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.596304] LustreError: 172106:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5788.607574] LustreError: 172106:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5788.632182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.837483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.099538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.411725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.612463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.033902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.128891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.297663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.564838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.953492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.439583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.632610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.217771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.609503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.021979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.632517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.469944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.259679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.175347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.724726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.834811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.832173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.863353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.212097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.433378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.635130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.183949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.224659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.134870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.887840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.095831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.154549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.880525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.792470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.118991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.159642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.954726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.855800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.964365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.797862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.671969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.646898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.743254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.893441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.747253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.662665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.479869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.439581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.458653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.509227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.416774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.428644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.555454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.679061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.746429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.918420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.879504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.813841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.928406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.849063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.076398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.078268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.169541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.926828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.713230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.551838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.251616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.988527] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5863.993375] Lustre: Skipped 235 previous similar messages [ 5864.003671] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5864.013325] Lustre: Skipped 235 previous similar messages [ 5864.017753] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5864.024522] Lustre: Skipped 235 previous similar messages [ 5864.028387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.038557] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5864.045312] Lustre: Skipped 235 previous similar messages [ 5864.921494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.892130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.836277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.819721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.186044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.261821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.304831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.434763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.724950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.915770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.121929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.322627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.797872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.776374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.926741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.348088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.499904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.195464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.134939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.158427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.192172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.260968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.472633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.355719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.500817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.692520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.554294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.808873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.002328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.251268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.398492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.437593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.681425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.867622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.983108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.147321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.181661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.219755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.056926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.948141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.743805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.665085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.573084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.461118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.268350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.169396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.130669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.258960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.283508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.250848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.254211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.249536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.534978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.331604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.212340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.023902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.886478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.803024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.817358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.698047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.561274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.516053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.449908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.253960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.218347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.140817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.038905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.903455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.943668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.841430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.992684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.072173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.223083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.655841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.032437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.805579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.404826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.116597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.993328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.838596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.791775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.768864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.664714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.472308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.517407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.533402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.281269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.400531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.557178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.567957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.515448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.400629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.470473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.408418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.287680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.235255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.086047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.934210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.847873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.712611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.725900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.542661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.676525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.558286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.455949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.411747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.640433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.763543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.883072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.123585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.106055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.109434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.042123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.031328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.782628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.779630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.699363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.513477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.392146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.371992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.347838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.346625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.273596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.261450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.287759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.230222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.200412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.216576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.241080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.048667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.826324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.708326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.537171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.430788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.367905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.325044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.170966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.122097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.941261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.969647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.946545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.990442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.928043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.755486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.697426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.534489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.378373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.196964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.012630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.908527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.692319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.599161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.459521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.442166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.383612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.518544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.480872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.855582] LustreError: 172106:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6019.867895] LustreError: 172106:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6019.889400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.887800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.449185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.044456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.443837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.438226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.842262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.259195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.220865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.739756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.160589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.620893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.195678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.474051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.659826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.680993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.890948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.944977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.955768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.408512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.874426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.795840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.263286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.476276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.500376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.160740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.405299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.322862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.420524] LustreError: 172106:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6055.429261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.430074] LustreError: 172106:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6056.338841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.988577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.826670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.823645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.952628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.789442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.891225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.927995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.276105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.507930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.625740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.563508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.724617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.851573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.225399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.917058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.723606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.453233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.507970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.345194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.340309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.149033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.979633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.767382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.653163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.383615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.211074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.089707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.064781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.039854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.958626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.819292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.779788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.632912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.574812] LustreError: 172106:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6088.585933] LustreError: 172106:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6088.592600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.354320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.327798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.164266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.240429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.267300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.310238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.287958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.344048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.274590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.139994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.099382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.971726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.928874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.838875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.906590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.652356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.687950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.469962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.578865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.462287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.574240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.513583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.532958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.619560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.760840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.801850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.848396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.751352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.700895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.677604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.776174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.789155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.023527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.331253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.459676] LustreError: 172106:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6123.472814] LustreError: 172106:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6123.482445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.580686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.830778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.058748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.393270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6129.564539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.752827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.976036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.930399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.314853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.487655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.493701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.833609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.915896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.914076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.203058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.123246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.442371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.651631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.713245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.816572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.157620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.342252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6150.336114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.208767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.334081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6153.426645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6154.444421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.665302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.623085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6157.796861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6158.776212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6159.996931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.010143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.009866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6163.359164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6164.703906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.095828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6167.236940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6168.213950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6169.328641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6170.449284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6171.437945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6173.073080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.515122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6175.877574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.000244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6178.398147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6179.381404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6180.746987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6182.216343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.564115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6185.438850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6186.771812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.428102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6189.609041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.132205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.943900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6192.828309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.061921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.472638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.294543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.312418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.534146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6199.528236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6200.558913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6201.653406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6202.545442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.665336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6204.516038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6205.389171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.306729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6207.403919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6208.511337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6209.833064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6210.984759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6212.343311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6213.363699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6214.546405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6215.668876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6216.895114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6218.032884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6219.485963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6220.387830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6221.664730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6222.987578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6224.479920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6225.575662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6227.069808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6228.204617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6229.432800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6230.299683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6231.204300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6232.326958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6233.529098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6234.899588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6236.225816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6237.589674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6239.185207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6240.414388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6241.819435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6242.753363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6243.855908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6245.299423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6246.733881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6248.422933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6249.544284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6250.795887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6252.144824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6253.057543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6254.455810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6255.238424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6256.423454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6257.697986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6259.123834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6260.300392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6261.553338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6262.523742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6263.730682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6264.720444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6265.814891] LustreError: 172106:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6265.827866] LustreError: 172106:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6265.834654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6266.816436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6267.865862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6268.909985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6269.865316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6270.903055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6271.923158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6273.045408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6273.988321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6275.089948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6276.304519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6277.346467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6278.543512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6279.642700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6280.732447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6281.841967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6282.871084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6284.167087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6285.095242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6286.150240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6287.005025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6288.107984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6289.168385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6290.340903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6291.436958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6292.599462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6293.690316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6294.494739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6295.575334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6296.859173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6297.816771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6299.041251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6299.957737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6300.940336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6302.150720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6303.334486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6304.713453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6306.229588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6307.749270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6308.828092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6310.115552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6311.298053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6312.266661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6313.359313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6314.275141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6315.400187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6316.335674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6317.345137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6318.456374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6319.594506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6320.505461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6321.490301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6322.464477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6323.610514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6324.486394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6325.709580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6326.671166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6327.789550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6328.660171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6329.744162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6330.870698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6331.839340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6332.840314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6333.774722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6335.078530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6336.036545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6337.106728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6338.232984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6339.345622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6340.311233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6341.650065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6342.679445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6344.248590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6345.592601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6346.972829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6348.345694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6349.702505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6351.168154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6352.107408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6353.343714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6353.999418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6354.934692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6356.210544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6357.440697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6358.916256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6360.266689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6361.678971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6363.084572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6364.101433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6365.066580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6366.231708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6367.451640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6368.786545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6370.127336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6371.580277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6373.109744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6374.340532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6375.564558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6376.572393] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6376.585238] Lustre: Skipped 473 previous similar messages [ 6376.611384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6376.624692] Lustre: Skipped 473 previous similar messages [ 6376.635957] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6376.645827] Lustre: Skipped 473 previous similar messages [ 6376.652236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6376.678493] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6376.688813] Lustre: Skipped 473 previous similar messages [ 6377.872101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6378.807685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6379.903987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6381.216452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6382.516650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6383.596778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6384.842789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6386.080812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6387.119821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6388.286627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6389.488813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6390.411911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6391.649700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6392.983293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6393.979659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6394.982835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6396.077312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6397.231440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6398.557064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6399.631302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6400.884482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6402.173789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6403.287436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6404.426832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6405.801446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6406.945253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6408.116889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6409.185653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6410.346817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6411.335668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6412.424810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6413.425551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6414.476523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6415.441828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6416.578587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6417.675217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6418.744328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6419.826755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6420.817966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6421.903246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6422.973351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6902.541946] LustreError: 173911:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6902.554416] LustreError: 172106:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 6902.566719] LustreError: 172106:0:(osp_precreate.c:655:osp_precreate_send()) Skipped 1 previous similar message [ 7596.003411] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7596.020183] Lustre: Skipped 41 previous similar messages [ 7596.031171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7596.041185] Lustre: Skipped 1 previous similar message [ 7599.031155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7599.048662] Lustre: Skipped 1 previous similar message [ 7602.107638] Lustre: server umount lustre-OST0000 complete [ 7608.805876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7608.819884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7613.082989] Lustre: server umount lustre-MDT0000 complete [ 7614.434090] LustreError: 173094:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7614.446628] LustreError: 173094:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7616.482537] LustreError: 172821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858286 with bad export cookie 587001839375225982 [ 7616.489170] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7616.496475] LustreError: 172821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7616.898557] Lustre: server umount lustre-MDT0001 complete [ 7626.212408] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 7629.172318] Key type lgssc unregistered [ 7629.578884] LNet: 227386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7629.590411] LNetError: 227386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7629.611954] LNet: Removed LNI 192.168.203.150@tcp [ 7630.476390] Key type .llcrypt unregistered [ 7630.479160] Key type ._llcrypt unregistered [ 7656.758801] Key type ._llcrypt registered [ 7656.760934] Key type .llcrypt registered [ 7656.835734] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 7657.843165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7657.882669] alg: No test for adler32 (adler32-zlib) [ 7659.182734] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 7659.493991] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 7661.232303] Key type lgssc registered [ 7662.472831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7671.292438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7671.308924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7672.601400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7677.537532] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7683.044992] LustreError: 228799:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7683.062155] LustreError: 228799:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7686.098738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7686.630271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7691.493527] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7699.650620] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7704.770492] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7711.030261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7711.503325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7717.745510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7717.971203] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7725.593111] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7731.129185] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7732.696325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7732.711418] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7732.716745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7732.724224] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7732.724977] LustreError: 228818:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7738.345177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7738.346223] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7738.351369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7738.369691] Lustre: Skipped 1 previous similar message [ 7740.369405] Lustre: server umount lustre-OST0000 complete [ 7752.752744] LustreError: 228818:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7773.975591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7774.229165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7778.357994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7778.388772] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7778.401439] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7778.425950] Lustre: Skipped 1 previous similar message [ 7780.909780] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7790.195908] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7792.226658] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7798.556333] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e044980e800.ost_server_uuid 50 [ 7800.614490] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9e044980e800.ost_server_uuid in FULL state after 0 sec [ 7809.507644] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7809.508812] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7809.533755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7809.570429] Lustre: Skipped 3 previous similar messages [ 7812.968785] Lustre: server umount lustre-OST0000 complete [ 7819.746533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7819.748525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7819.777959] Lustre: Skipped 1 previous similar message [ 7819.782945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7819.824844] Lustre: Skipped 1 previous similar message [ 7824.513835] Lustre: server umount lustre-MDT0000 complete [ 7824.866522] LustreError: 228793:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7824.883473] LustreError: 228793:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7827.978980] LustreError: 232032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858497 with bad export cookie 15278451850746177653 [ 7827.980865] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7827.992575] LustreError: 232032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7828.262665] Lustre: server umount lustre-MDT0001 complete [ 7840.190257] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 7842.959606] Key type lgssc unregistered [ 7843.263559] LNet: 234009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7843.270294] LNetError: 234009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7843.289146] LNet: Removed LNI 192.168.203.150@tcp [ 7844.236133] Key type .llcrypt unregistered [ 7844.238192] Key type ._llcrypt unregistered [ 7876.924935] Key type ._llcrypt registered [ 7876.926574] Key type .llcrypt registered [ 7877.113412] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 7878.484984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7878.508083] alg: No test for adler32 (adler32-zlib) [ 7879.565765] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 7879.827763] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 7881.544220] Key type lgssc registered [ 7882.613731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7892.297360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7892.317447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7893.947999] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7898.861492] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7904.235370] LustreError: 235735:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7904.262029] LustreError: 235735:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7907.863129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7908.296496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7912.562141] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7915.759587] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7922.916873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7923.307427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7929.713870] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7938.676055] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7938.683262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7938.949975] Lustre: lustre-OST0001: new disk, initializing [ 7938.954178] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7938.964072] Lustre: 238042:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7939.081236] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7946.134574] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7946.395965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7946.412815] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7946.621710] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7957.933603] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7966.352335] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7977.920463] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7977.924057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7977.969646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7980.181231] Lustre: server umount lustre-MDT0000 complete [ 7982.563611] LustreError: 235735:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7982.569429] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7982.604163] Lustre: Skipped 2 previous similar messages [ 7983.883130] LustreError: 235717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858653 with bad export cookie 18166101717089754626 [ 7983.886561] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7983.892290] LustreError: 235717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7984.236824] Lustre: server umount lustre-MDT0001 complete [ 7998.130823] Lustre: server umount lustre-OST0000 complete [ 8012.286131] Lustre: server umount lustre-OST0001 complete [ 8034.772615] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 8044.461914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8044.941975] LustreError: 241151:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8044.969844] LustreError: 241151:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8045.042967] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8049.690952] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8050.150623] LustreError: 241152:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8050.162161] LustreError: 241152:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8055.267662] LustreError: 241151:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8058.165695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8058.600988] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8063.760916] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8067.499701] Lustre: 242294:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8075.173664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8075.656921] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8082.319423] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8083.907411] LustreError: 242650:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8083.932423] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 8090.119357] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49760) [ 8091.132974] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8096.778279] Lustre: mylustre-OST0001: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33) [ 8098.356342] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8105.658881] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8110.497667] Lustre: 244165:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8125.413217] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8125.437270] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8125.463456] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8127.464341] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8127.478196] Lustre: mylustre-MDT0000-lwp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8127.506839] Lustre: Skipped 1 previous similar message [ 8128.935710] Lustre: server umount mylustre-MDT0000 complete [ 8132.581959] LustreError: 241152:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8132.606381] LustreError: 241152:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 8132.873347] LustreError: 241128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858802 with bad export cookie 18166101717089757489 [ 8132.881261] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8132.893915] LustreError: 241128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8133.343516] Lustre: server umount mylustre-MDT0001 complete [ 8148.077989] Lustre: server umount mylustre-OST0000 complete [ 8161.673413] Lustre: server umount mylustre-OST0001 complete [ 8188.261550] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 8198.952771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8199.397113] LustreError: 246475:0:(ldlm_lib.c:1199:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8199.473099] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8199.478639] Lustre: Skipped 1 previous similar message [ 8203.748894] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8211.873072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8216.569832] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8219.788069] Lustre: 247616:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8226.630712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8234.544516] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8237.313206] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:161) [ 8244.273334] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8245.624533] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49792) [ 8245.631602] Lustre: tfs-OST0001: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:65) [ 8251.217979] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8259.711723] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8263.578489] Lustre: 249489:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8271.332668] Lustre: tfs-MDT0000-lwp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8271.344959] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 8271.361532] Lustre: Skipped 4 previous similar messages [ 8271.369760] Lustre: Skipped 6 previous similar messages [ 8276.454871] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 8276.783317] Lustre: server umount tfs-MDT0000 complete [ 8279.958304] LustreError: 246452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858949 with bad export cookie 18166101717089759659 [ 8279.963143] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8279.968428] LustreError: 246452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8280.366338] Lustre: server umount tfs-MDT0001 complete [ 8294.825310] Lustre: server umount tfs-OST0000 complete [ 8297.954660] Lustre: 234996:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788858951/real 1788858951] req@ffff9ff25e108000 x1875754142089088/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788858967 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8298.024256] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8299.579930] Lustre: server umount tfs-OST0001 complete [ 8323.737951] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 8334.334678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8334.925461] LustreError: 251739:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8334.944842] LustreError: 251739:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 8335.024037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8335.032096] Lustre: Skipped 3 previous similar messages [ 8340.365294] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8350.528854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8355.831298] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8359.016623] Lustre: 252880:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8366.465299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8372.868436] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8376.218422] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:193) [ 8380.293437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49824) [ 8382.334319] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8388.146120] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:97) [ 8391.390100] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8405.922807] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8410.742729] Lustre: 254753:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8420.082732] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 05:18:07 (1788859087) [ 8438.243729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8438.249135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8438.259394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8438.266524] Lustre: Skipped 3 previous similar messages [ 8444.583369] Lustre: server umount lustre-MDT0000 complete [ 8448.399631] LustreError: 253259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859118 with bad export cookie 18166101717089761584 [ 8448.411543] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8448.418780] LustreError: 253259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8454.870253] Lustre: server umount lustre-MDT0001 complete [ 8459.974720] Lustre: server umount lustre-OST0000 complete [ 8464.535253] Lustre: server umount lustre-OST0001 complete [ 8472.892809] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 8487.925936] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 8527.515347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8527.886455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8527.912541] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8528.003578] Lustre: lustre-MDT0000: new disk, initializing [ 8528.130035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 8528.137219] Lustre: Skipped 3 previous similar messages [ 8528.163580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8533.228186] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8544.882721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8545.099230] Lustre: 257944:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8545.200657] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8545.204805] Lustre: Skipped 1 previous similar message [ 8545.286367] Lustre: lustre-MDT0001: new disk, initializing [ 8545.566209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8545.589655] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8551.077682] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8562.261210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8562.650915] Lustre: lustre-OST0000: new disk, initializing [ 8562.663224] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8562.690543] Lustre: 259525:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8564.649036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8564.660764] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8564.833981] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8569.744965] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8581.656031] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8581.827537] Lustre: lustre-OST0001: new disk, initializing [ 8581.830858] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8581.839911] Lustre: 260398:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8583.338033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8583.353177] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8583.499387] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8588.560739] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8600.954349] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 05:21:08 (1788859268) [ 8617.619038] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 05:21:25 (1788859285) [ 8634.860377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8634.870000] Lustre: Skipped 6 previous similar messages [ 8634.879980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8634.889772] Lustre: Skipped 13 previous similar messages [ 8639.218812] Lustre: server umount lustre-OST0000 complete [ 8639.977787] LustreError: 259508:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8639.996083] LustreError: 259508:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 8649.889375] Lustre: server umount lustre-MDT0000 complete [ 8653.459962] LustreError: 259524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859323 with bad export cookie 18166101717089763362 [ 8653.466164] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8653.470723] LustreError: 259524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8659.771683] Lustre: server umount lustre-MDT0001 complete [ 8670.850573] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 8745.936379] Lustre: server umount lustre-OST0001 complete [ 8753.559992] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 8767.942754] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 8804.352703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8804.832690] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8804.882779] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8805.017687] Lustre: lustre-MDT0000: new disk, initializing [ 8805.188339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8805.196384] Lustre: Skipped 3 previous similar messages [ 8805.209720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8810.491688] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8825.015282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8825.258076] Lustre: 265252:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8825.371792] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8825.379140] Lustre: Skipped 1 previous similar message [ 8825.591249] Lustre: lustre-MDT0001: new disk, initializing [ 8825.840585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8825.856861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8831.866854] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8844.456454] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8851.664785] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8859.863588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8860.190352] Lustre: lustre-OST0000: new disk, initializing [ 8860.193837] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8860.199155] Lustre: 267223:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8861.605392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8861.614736] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8861.772879] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8867.359474] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8878.984315] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8892.390175] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8892.394922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8892.416557] LustreError: Skipped 1 previous similar message [ 8892.417334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8892.417337] Lustre: Skipped 5 previous similar messages [ 8892.463168] Lustre: Skipped 5 previous similar messages [ 8894.262762] Lustre: server umount lustre-OST0000 complete [ 8902.626352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8907.750929] LustreError: 267307:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8907.783027] LustreError: 267307:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 8907.980769] Lustre: server umount lustre-MDT0000 complete [ 8913.259104] LustreError: 267222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859582 with bad export cookie 18166101717089764832 [ 8913.266178] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8913.269286] LustreError: 267222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8913.793714] Lustre: server umount lustre-MDT0001 complete [ 8925.920382] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 8928.918326] Key type lgssc unregistered [ 8929.196820] LNet: 268987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8929.205484] LNetError: 268987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8929.234343] LNet: Removed LNI 192.168.203.150@tcp [ 8930.071237] Key type .llcrypt unregistered [ 8930.075755] Key type ._llcrypt unregistered [ 8958.581197] Key type ._llcrypt registered [ 8958.590185] Key type .llcrypt registered [ 8958.726716] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 8959.873645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8960.002236] alg: No test for adler32 (adler32-zlib) [ 8961.125051] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 8961.386130] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 8963.072222] Key type lgssc registered [ 8964.085184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8971.431909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8971.467439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8973.099178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8977.256441] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8982.517979] LustreError: 270445:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8982.553658] LustreError: 270445:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8987.307026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8987.618115] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8987.867484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8992.592802] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9000.373982] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9006.446561] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9015.149335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9015.764235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9017.867263] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 9023.021630] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9031.236473] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9032.295325] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 9038.313459] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9038.332259] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9038.349561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9040.711633] Lustre: server umount lustre-OST0000 complete [ 9048.548053] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9048.553887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9048.556159] Lustre: Skipped 2 previous similar messages [ 9048.574426] Lustre: Skipped 2 previous similar messages [ 9050.768984] Lustre: server umount lustre-MDT0000 complete [ 9053.675356] LustreError: 270439:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9053.714388] LustreError: 270439:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9055.072697] LustreError: 270424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859724 with bad export cookie 11118167575366376688 [ 9055.077375] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9055.095734] LustreError: 270424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9055.627318] Lustre: server umount lustre-MDT0001 complete [ 9069.812409] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 9074.139571] Key type lgssc unregistered [ 9074.466144] LNet: 274029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9074.472728] LNetError: 274029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9074.480561] LNet: Removed LNI 192.168.203.150@tcp [ 9075.345691] Key type .llcrypt unregistered [ 9075.348337] Key type ._llcrypt unregistered [ 9093.338744] Key type ._llcrypt registered [ 9093.341303] Key type .llcrypt registered [ 9093.499043] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9094.502819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9094.601683] alg: No test for adler32 (adler32-zlib) [ 9095.821865] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 9096.081604] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 9097.745504] Key type lgssc registered [ 9099.395451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9109.760325] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9109.789220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9125.344566] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9130.466845] LustreError: 275220:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.150@tcp: failed processing log, type 4: rc = -110 [ 9157.217047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9164.669699] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9172.726764] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9175.209947] Lustre: server umount lustre-OST0000 complete [ 9193.096691] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 9196.733510] Key type lgssc unregistered [ 9197.182814] LNet: 276850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9197.206498] LNetError: 276850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9197.294070] LNet: Removed LNI 192.168.203.150@tcp [ 9198.916722] Key type .llcrypt unregistered [ 9198.918375] Key type ._llcrypt unregistered [ 9248.505478] Key type ._llcrypt registered [ 9248.511540] Key type .llcrypt registered [ 9248.708937] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9250.600845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9250.619075] alg: No test for adler32 (adler32-zlib) [ 9251.734339] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 9252.044401] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 9253.680238] Key type lgssc registered [ 9254.830174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9268.191057] loop0: detected capacity change from 0 to 204800000 [ 9268.404524] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9268.430518] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9268.693726] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9269.734643] loop0: detected capacity change from 0 to 204800000 [ 9269.879405] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9269.890071] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9270.207807] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9271.241836] loop0: detected capacity change from 0 to 204800000 [ 9271.388985] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9271.402622] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9271.514836] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9272.408695] loop0: detected capacity change from 0 to 204800000 [ 9272.565417] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9273.625966] loop0: detected capacity change from 0 to 204800000 [ 9273.650514] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9277.760800] loop0: detected capacity change from 0 to 204800000 [ 9277.807662] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9281.412039] loop0: detected capacity change from 0 to 204800000 [ 9281.441716] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9284.598943] loop0: detected capacity change from 0 to 204800000 [ 9284.633242] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9288.357304] loop0: detected capacity change from 0 to 204800000 [ 9288.419166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9288.439259] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9288.499188] Lustre: 280268:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9288.507460] Lustre: 280268:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 9288.516965] Lustre: 280268:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 9288.529493] Lustre: 280268:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9288.544856] Lustre: 280268:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9288.553695] Lustre: 280268:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 9288.559872] Lustre: 280268:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 9288.567630] Lustre: 280268:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9288.572882] Lustre: 280268:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 9288.580376] Lustre: 280268:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 9288.585918] Lustre: 280268:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 9288.599645] Lustre: 280268:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 9288.605226] Lustre: 280268:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 9288.611908] Lustre: 280268:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 9295.766638] Lustre: server umount MGS complete [ 9296.532671] loop0: detected capacity change from 0 to 204800000 [ 9296.654332] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9297.375668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9297.460410] Lustre: lustre-MDT0000: Aborting client recovery [ 9298.747652] loop1: detected capacity change from 0 to 204800000 [ 9298.951747] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9299.113654] Lustre: 280700:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9299.128065] Lustre: 280700:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 9299.139611] Lustre: 280700:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 9299.166618] Lustre: 280700:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9299.183343] Lustre: 280700:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 9299.196608] Lustre: 280700:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9299.215212] Lustre: 280700:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9299.286510] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9299.777830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9299.853581] Lustre: lustre-MDT0001: Aborting client recovery [ 9301.143608] loop2: detected capacity change from 0 to 204800000 [ 9301.318642] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9301.480809] Lustre: 280781:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9301.529567] Lustre: 280781:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9301.553371] Lustre: 280781:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9301.578306] Lustre: 280781:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9301.598183] Lustre: 280781:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9301.624579] Lustre: 280781:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9301.657707] Lustre: 280781:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9301.868828] Lustre: Found index 0 for lustre-OST0000, updating log [ 9301.982345] Lustre: 280800:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9302.163183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9302.214779] Lustre: lustre-OST0000: Aborting recovery [ 9303.392577] loop3: detected capacity change from 0 to 204800000 [ 9303.588551] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9303.636387] Lustre: 280865:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9303.645508] Lustre: 280865:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9303.662857] Lustre: 280865:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9303.678620] Lustre: 280865:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9303.698457] Lustre: 280865:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9303.706023] Lustre: 280865:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9303.714995] Lustre: 280865:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9303.731402] Lustre: Found index 1 for lustre-OST0001, updating log [ 9303.750533] Lustre: 280875:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9303.829677] Lustre: lustre-OST0001: Aborting recovery [ 9306.943879] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9306.964205] LustreError: 280644:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -108 [ 9307.091989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 9307.945263] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9308.063555] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 9309.999586] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 9310.004237] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 9313.568330] LustreError: 280611:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 1999us: rc = -2 [ 9314.080827] LustreError: 280609:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 457us: rc = -2 [ 9314.109770] LustreError: 280609:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 20 previous similar messages [ 9330.657648] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9330.657988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9330.660672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9330.679744] Lustre: Skipped 1 previous similar message [ 9333.267847] Lustre: server umount lustre-MDT0000 complete [ 9334.005677] LustreError: 280593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788860003 with bad export cookie 12736348574608097729 [ 9334.015270] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9334.028354] LustreError: 280593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9335.781093] LustreError: 281155:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9335.782703] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9335.784043] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9335.784049] Lustre: Skipped 3 previous similar messages [ 9335.791785] LustreError: 281155:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9335.818676] Lustre: Skipped 3 previous similar messages [ 9339.809361] LustreError: 280610:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9339.809632] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9339.852785] Lustre: Skipped 2 previous similar messages [ 9340.520904] Lustre: server umount lustre-MDT0001 complete [ 9347.795790] Lustre: server umount lustre-OST0000 complete [ 9355.122413] Lustre: server umount lustre-OST0001 complete [ 9363.552889] Lustre: DEBUG MARKER: == conf-sanity test 108c: mkfs.lustre --mmp option enables MMP feature ========================================================== 05:33:51 (1788860031) [ 9385.594318] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9427.498942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9427.801713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9427.863505] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9427.987138] Lustre: lustre-MDT0000: new disk, initializing [ 9428.099502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9428.107273] Lustre: Skipped 1 previous similar message [ 9428.136260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9432.940454] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9439.460608] Lustre: server umount lustre-MDT0000 complete [ 9456.570765] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 05:35:24 (1788860124) [ 9487.373906] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 9500.061503] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9543.403580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9543.788709] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9543.822716] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9543.828610] Lustre: Skipped 1 previous similar message [ 9543.919852] Lustre: lustre-MDT0000: new disk, initializing [ 9543.993286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9544.016747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9549.947397] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9563.385475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9563.531410] Lustre: 287978:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9563.621743] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9563.628956] Lustre: Skipped 1 previous similar message [ 9563.757922] Lustre: lustre-MDT0001: new disk, initializing [ 9563.855579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9563.908396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9563.928594] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9569.072868] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9580.028122] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9586.272505] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9593.647946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9593.931615] Lustre: lustre-OST0000: new disk, initializing [ 9593.935389] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9593.948447] Lustre: 289948:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9594.064403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9600.926287] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9603.650454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9603.677607] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9603.770638] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9611.069917] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9616.796314] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9627.536627] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9634.221182] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9646.383638] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9696.227909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9696.230972] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9696.238272] LustreError: Skipped 1 previous similar message [ 9696.239783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9696.260715] Lustre: Skipped 1 previous similar message [ 9697.429313] Lustre: server umount lustre-OST0000 complete [ 9701.346559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9701.353828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9701.362279] Lustre: Skipped 1 previous similar message [ 9701.379108] Lustre: Skipped 2 previous similar messages [ 9707.666496] Lustre: server umount lustre-MDT0000 complete [ 9711.585141] LustreError: 287985:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9711.602749] LustreError: 287985:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9711.931312] LustreError: 291148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788860381 with bad export cookie 12736348574608100802 [ 9711.933767] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9711.949281] LustreError: 291148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9712.253088] Lustre: server umount lustre-MDT0001 complete [ 9721.871806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9726.775133] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9733.244967] Lustre: server umount MGS complete [ 9748.160871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9748.565293] LustreError: 293791:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9748.579323] LustreError: 293804:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9748.617570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9754.087691] LustreError: 293792:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9754.625632] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9764.322694] LustreError: 293792:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9764.348800] LustreError: 293792:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9766.106087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9771.204142] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9779.577828] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9786.011957] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9792.070357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9797.922980] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9804.539725] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9824.229592] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9824.241941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9824.274653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9824.285181] Lustre: Skipped 2 previous similar messages [ 9827.311543] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9830.045531] Lustre: server umount lustre-OST0000 complete [ 9837.537380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9837.547324] Lustre: Skipped 2 previous similar messages [ 9842.658462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9842.667671] Lustre: Skipped 5 previous similar messages [ 9848.288562] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9848.438538] Lustre: server umount lustre-MDT0000 complete [ 9851.912862] LustreError: 293772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788860521 with bad export cookie 12736348574608102650 [ 9851.926536] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9851.930521] LustreError: 293772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9852.332402] Lustre: server umount lustre-MDT0001 complete [ 9865.733801] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 9869.805352] Key type lgssc unregistered [ 9870.118758] LNet: 297732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9870.126286] LNetError: 297732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9870.157453] LNet: Removed LNI 192.168.203.150@tcp [ 9871.172926] Key type .llcrypt unregistered [ 9871.175353] Key type ._llcrypt unregistered [ 9908.187671] Key type ._llcrypt registered [ 9908.189855] Key type .llcrypt registered [ 9908.305099] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 9924.698058] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9926.310314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9926.394938] alg: No test for adler32 (adler32-zlib) [ 9927.569924] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 9927.881972] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 9929.577639] Key type lgssc registered [ 9931.495379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9972.563477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9972.615238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9973.941260] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9973.989633] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9974.070310] Lustre: lustre-MDT0000: new disk, initializing [ 9974.139225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9974.151375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9980.464194] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9993.332738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9993.519547] Lustre: 301690:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9993.601526] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9993.605749] Lustre: Skipped 1 previous similar message [ 9993.755813] Lustre: lustre-MDT0001: new disk, initializing [ 9993.950250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9994.073973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9994.092805] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9999.006890] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10009.342543] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10014.841439] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10020.728591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10021.067204] Lustre: lustre-OST0000: new disk, initializing [10021.070634] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10021.081346] Lustre: 303658:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10021.174700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10027.063170] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10029.173639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10029.184715] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10029.286417] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10036.399546] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10043.492285] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [10056.618954] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [10062.185572] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [10111.458484] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10111.470985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10111.477276] Lustre: Skipped 1 previous similar message [10111.482357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10111.515230] Lustre: Skipped 1 previous similar message [10116.580302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10116.587843] Lustre: Skipped 1 previous similar message [10117.838812] Lustre: server umount lustre-OST0000 complete [10122.209123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10122.223154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10122.248457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10126.819641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10126.820818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10126.846929] Lustre: Skipped 1 previous similar message [10128.180941] Lustre: server umount lustre-MDT0000 complete [10132.062641] LustreError: 301682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788860801 with bad export cookie 1738141569077864463 [10132.063500] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10132.069878] LustreError: 301682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10132.351711] Lustre: server umount lustre-MDT0001 complete [10141.227103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10146.227961] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10151.433726] Lustre: server umount MGS complete [10163.016224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10163.461740] LustreError: 307571:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10163.489561] LustreError: 307584:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [10163.503842] LustreError: 307571:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10163.568594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10168.267636] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10168.812828] LustreError: 307572:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10173.923581] LustreError: 307571:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10176.724510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10177.200172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10183.169364] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10191.796090] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10197.780682] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10204.308246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10204.789468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10212.146400] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10221.477506] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10245.089685] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10245.094297] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10245.112861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10251.226626] Lustre: server umount lustre-OST0000 complete [10258.415925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10258.423858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10258.443354] Lustre: Skipped 2 previous similar messages [10258.471415] Lustre: Skipped 3 previous similar messages [10263.276164] Lustre: server umount lustre-MDT0000 complete [10263.531930] LustreError: 307571:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10263.569781] LustreError: 307571:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10267.309846] LustreError: 308306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788860937 with bad export cookie 1738141569077866318 [10267.310123] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10267.319360] LustreError: 308306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10267.702115] Lustre: server umount lustre-MDT0001 complete [10279.609590] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [10282.549715] Key type lgssc unregistered [10283.030681] LNet: 311513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10283.048442] LNetError: 311513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10283.089354] LNet: Removed LNI 192.168.203.150@tcp [10284.515516] Key type .llcrypt unregistered [10284.521374] Key type ._llcrypt unregistered [10310.372772] Key type ._llcrypt registered [10310.379558] Key type .llcrypt registered [10310.518934] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [10327.754787] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [10329.515645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10329.592306] alg: No test for adler32 (adler32-zlib) [10330.685419] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [10330.908255] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [10332.656544] Key type lgssc registered [10334.710424] Lustre: Echo OBD driver; http://www.lustre.org/ [10368.843988] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [10384.431346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10384.473521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10385.688319] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10385.709543] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10385.763717] Lustre: lustre-MDT0000: new disk, initializing [10385.842810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10385.866946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10390.201190] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10402.813797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10403.025251] Lustre: lustre-OST0000: new disk, initializing [10403.027817] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10403.030276] Lustre: Skipped 1 previous similar message [10403.033317] Lustre: 315564:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10403.083492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10411.024101] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10411.113696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10411.121395] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10411.147738] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11337.706386] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11337.723398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11340.628603] Lustre: server umount lustre-MDT0000 complete [11344.370837] LustreError: 314748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788862014 with bad export cookie 13927946931253894584 [11344.376330] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11344.386232] LustreError: 314748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11344.583366] Lustre: server umount lustre-OST0000 complete [11370.937206] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [11380.998548] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [11415.090413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11415.298694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11415.322102] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11415.376246] Lustre: lustre-MDT0000: new disk, initializing [11415.459173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11415.481991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11419.775387] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11429.339274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11429.473263] Lustre: 319927:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11429.550746] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11429.556651] Lustre: Skipped 1 previous similar message [11429.642914] Lustre: lustre-MDT0001: new disk, initializing [11429.744808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11429.775548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11429.786597] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11434.096711] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11442.713594] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11447.704256] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11453.659518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11453.937445] Lustre: lustre-OST0000: new disk, initializing [11453.941387] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11453.947149] Lustre: 321895:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11454.011653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11459.737455] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11463.266884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11463.286142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11463.433044] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11469.948603] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11476.046584] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11476.313657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11481.005089] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11481.277565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11484.129543] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11484.134064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11484.148057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11484.163276] Lustre: Skipped 1 previous similar message [11484.175687] Lustre: Skipped 1 previous similar message [11489.251212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11489.266551] Lustre: Skipped 1 previous similar message [11489.707507] Lustre: server umount lustre-OST0000 complete [11494.373374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11494.387374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11494.390652] Lustre: Skipped 1 previous similar message [11499.489719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11499.497255] Lustre: Skipped 1 previous similar message [11499.896580] Lustre: server umount lustre-MDT0000 complete [11504.520370] LustreError: 320723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788862174 with bad export cookie 13927946931256835403 [11504.529109] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11504.533965] LustreError: 320723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11504.824660] Lustre: server umount lustre-MDT0001 complete [11508.829341] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [11510.623855] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 06:09:38 (1788862178) [11516.647234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11517.111150] LustreError: 324206:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11517.129501] LustreError: 324206:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11517.175112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11521.218904] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11522.531492] LustreError: 324207:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11526.631827] LustreError: 324206:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11528.481892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11528.761563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11533.001315] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11540.071405] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11545.024231] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11550.804482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11551.077308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11556.461291] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11564.356207] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11572.182477] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11572.469676] Lustre: lustre-OST0001: new disk, initializing [11572.476581] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11572.498948] Lustre: 327042:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11572.603531] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11577.988883] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11579.985252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11579.993913] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11580.049760] LustreError: 327503:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11581.089684] LustreError: 327503:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11581.100407] LustreError: 327503:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [11582.126370] LustreError: 327503:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11584.228193] LustreError: 327503:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11584.237953] LustreError: 327503:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [11586.156772] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11586.302656] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11588.323927] LustreError: 327507:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11588.336968] LustreError: 327507:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 6 previous similar messages [11590.768618] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e0444f78000.ost_server_uuid 50 [11592.046473] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9e0444f78000.ost_server_uuid in FULL state after 0 sec [11596.587768] LustreError: 327503:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11596.600687] LustreError: 327503:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 16 previous similar messages [11597.095947] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e0444f78000.ost_server_uuid 50 [11598.359663] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9e0444f78000.ost_server_uuid in FULL state after 0 sec [11604.973966] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11612.995058] Lustre: lustre-OST0001: Not available for connect from 192.168.203.50@tcp (stopping) [11615.201474] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11615.209236] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11615.219053] LustreError: Skipped 2 previous similar messages [11615.245499] Lustre: Skipped 2 previous similar messages [11618.697578] Lustre: server umount lustre-OST0001 complete [11621.350976] LustreError: 326025:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11621.374356] LustreError: 326025:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11625.954988] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11625.965623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11626.483709] LustreError: 326026:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11626.498175] LustreError: 326026:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11630.619558] Lustre: server umount lustre-OST0000 complete [11636.196181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11636.203395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11636.212201] Lustre: Skipped 1 previous similar message [11636.215722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11636.218790] Lustre: Skipped 8 previous similar messages [11640.583089] Lustre: server umount lustre-MDT0000 complete [11641.827352] LustreError: 324933:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11641.835175] LustreError: 324933:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11644.119759] LustreError: 324189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788862313 with bad export cookie 13927946931256836285 [11644.130983] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11644.134043] LustreError: 324189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11644.568376] Lustre: server umount lustre-MDT0001 complete [11654.360452] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [11656.851470] Key type lgssc unregistered [11657.094879] LNet: 329731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11657.102500] LNetError: 329731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11657.117788] LNet: Removed LNI 192.168.203.150@tcp [11657.838471] Key type .llcrypt unregistered [11657.843248] Key type ._llcrypt unregistered [11678.311301] Key type ._llcrypt registered [11678.314621] Key type .llcrypt registered [11678.428299] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [11679.283196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11679.297383] alg: No test for adler32 (adler32-zlib) [11680.320839] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [11680.582698] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [11682.256600] Key type lgssc registered [11683.331857] Lustre: Echo OBD driver; http://www.lustre.org/ [11689.434938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11689.461312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11690.894890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11694.982181] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11701.219749] LustreError: 331127:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11701.248085] LustreError: 331127:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11701.893718] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11706.338665] LustreError: 331128:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11707.091606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11707.417363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11711.095668] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11718.573703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11718.870606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11723.882324] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11729.856792] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11733.167306] LustreError: 332809:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11734.814066] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11735.013805] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11736.064415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [11736.066474] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [11739.715699] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11745.526766] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11751.813478] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [11752.029525] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11853.802450] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11853.806239] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11853.809988] LustreError: Skipped 1 previous similar message [11853.817668] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11853.832400] Lustre: Skipped 1 previous similar message [11856.575683] Lustre: server umount lustre-OST0001 complete [11858.921140] LustreError: 332806:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11858.940335] LustreError: 332806:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11864.042727] LustreError: 332808:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11864.047843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11864.048500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11864.050258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11864.050265] Lustre: Skipped 1 previous similar message [11864.074107] LustreError: 332808:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11864.117824] Lustre: Skipped 1 previous similar message [11869.156331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11869.170883] Lustre: Skipped 2 previous similar messages [11874.292804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11874.296545] LustreError: 333681:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11874.299442] Lustre: Skipped 1 previous similar message [11874.315789] LustreError: 333681:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11876.320119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11876.505874] Lustre: server umount lustre-OST0000 complete [11881.446759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11881.453277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11881.468049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11884.519713] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11889.639925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11889.648607] Lustre: Skipped 3 previous similar messages [11894.240405] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11894.415654] Lustre: server umount lustre-MDT0000 complete [11894.759947] LustreError: 331122:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11894.793569] LustreError: 331122:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11897.639150] LustreError: 332060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788862567 with bad export cookie 307277742534161626 [11897.646230] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11897.655845] LustreError: 332060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11897.916627] Lustre: server umount lustre-MDT0001 complete [11906.394798] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [11908.874924] Key type lgssc unregistered [11909.155487] LNet: 336068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11909.166982] LNetError: 336068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11909.191688] LNet: Removed LNI 192.168.203.150@tcp [11909.964083] Key type .llcrypt unregistered [11909.966083] Key type ._llcrypt unregistered [11947.389412] Key type ._llcrypt registered [11947.391703] Key type .llcrypt registered [11947.517969] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [11948.427248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11948.554229] alg: No test for adler32 (adler32-zlib) [11949.624348] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [11949.829537] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [11951.544202] Key type lgssc registered [11952.571186] Lustre: Echo OBD driver; http://www.lustre.org/ [11960.368476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11960.388591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11961.783081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11965.588232] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11972.068859] LustreError: 338356:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11972.085442] LustreError: 338356:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11972.739344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11973.019537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11976.635320] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11978.998797] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11984.307637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11984.647170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11986.667960] LustreError: 339842:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11986.691514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [11989.762758] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11992.046340] LustreError: 340000:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11996.065915] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11996.261838] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11997.298039] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [12001.117988] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12009.595829] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12014.625428] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [12024.291222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12024.305130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12024.340043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12028.391278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12028.399309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12028.407781] Lustre: Skipped 2 previous similar messages [12030.348584] Lustre: server umount lustre-MDT0000 complete [12033.202816] LustreError: 338336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788862702 with bad export cookie 7576482217224922140 [12033.206202] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12033.217565] LustreError: 338336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12033.528891] Lustre: server umount lustre-MDT0001 complete [12046.412688] Lustre: server umount lustre-OST0000 complete [12049.377149] Lustre: 337618:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788862703/real 1788862703] req@ffff9ff35e470a80 x1875758409639168/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788862719 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12049.423680] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12050.741626] Lustre: server umount lustre-OST0001 complete [12065.849559] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [12075.126618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12075.518822] LustreError: 343618:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12075.586244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12079.952579] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12080.620106] LustreError: 343619:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12087.946116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12088.324192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12093.152573] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12096.222839] Lustre: 344758:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [12102.991133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12109.668869] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12112.552216] LustreError: 345112:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12112.593687] LustreError: 345112:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12118.581815] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12118.887353] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12118.894868] Lustre: Skipped 1 previous similar message [12120.951898] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [12120.955381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [12124.982845] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12132.065146] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12135.500306] Lustre: 346631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12139.491141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12139.510337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12139.539053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12139.552969] Lustre: Skipped 3 previous similar messages [12141.544081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12141.555151] Lustre: Skipped 1 previous similar message [12145.492771] Lustre: server umount lustre-MDT0000 complete [12146.659180] LustreError: 346033:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12146.675034] LustreError: 346033:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12149.081650] LustreError: 343599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788862818 with bad export cookie 7576482217224923841 [12149.083838] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12149.093601] LustreError: 343599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12149.683118] Lustre: server umount lustre-MDT0001 complete [12164.352504] Lustre: server umount lustre-OST0000 complete [12167.970714] Lustre: 337616:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788862821/real 1788862821] req@ffff9ff351633800 x1875758409687936/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788862837 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12168.003651] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12168.022804] Lustre: Skipped 3 previous similar messages [12168.208744] Lustre: server umount lustre-OST0001 complete [12171.712331] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [12173.249974] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [12174.951542] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 06:20:42 (1788862842) [12196.907721] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [12205.719233] loop0: detected capacity change from 0 to 1099511627776 [12206.380976] SGI XFS with ACLs, security attributes, quota, no debug enabled [12206.390621] *** VALIDATE xfs *** [12206.406837] XFS (loop0): Mounting V5 Filesystem [12206.437203] XFS (loop0): Ending clean mount [12207.531493] loop1: detected capacity change from 0 to 18691697672192 [12308.801977] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [12318.811520] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [12352.584291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12352.753872] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12352.785438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12352.888766] Lustre: lustre-MDT0000: new disk, initializing [12352.973117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12352.991586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12356.929338] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12366.621065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12366.782666] Lustre: 351764:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12366.867301] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12366.873092] Lustre: Skipped 1 previous similar message [12366.959156] Lustre: lustre-MDT0001: new disk, initializing [12367.134050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12367.144807] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12371.375188] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12380.185867] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12386.028168] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12393.552092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12393.842286] Lustre: lustre-OST0000: new disk, initializing [12393.846945] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12393.857091] Lustre: 353731:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12395.921269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12395.934943] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12396.094950] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12400.749133] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12409.367220] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12414.123099] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12414.388098] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12418.934702] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12419.248188] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12421.606715] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12421.611410] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12421.613235] LustreError: Skipped 1 previous similar message [12421.619973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12421.625224] Lustre: Skipped 1 previous similar message [12421.640867] Lustre: Skipped 1 previous similar message [12427.586920] Lustre: server umount lustre-OST0000 complete [12431.854761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12431.862382] Lustre: Skipped 3 previous similar messages [12437.466812] Lustre: server umount lustre-MDT0000 complete [12441.009270] LustreError: 351754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788863110 with bad export cookie 7576482217224925388 [12441.014972] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12441.020417] LustreError: 351754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12441.410786] Lustre: server umount lustre-MDT0001 complete [12448.695912] XFS (loop0): Unmounting Filesystem [12453.070716] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 06:25:20 (1788863120) [12459.063063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12459.415240] LustreError: 356291:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12459.480459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12459.482886] Lustre: Skipped 2 previous similar messages [12463.600325] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12471.772574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12475.990598] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12482.635890] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12487.595658] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12493.836254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12499.117381] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12505.524364] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12514.786652] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12514.788680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12514.800604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12514.811611] Lustre: Skipped 3 previous similar messages [12514.826321] Lustre: Skipped 3 previous similar messages [12519.300455] Lustre: server umount lustre-OST0000 complete [12523.489068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12529.766902] Lustre: server umount lustre-MDT0000 complete [12530.146676] LustreError: 357358:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12530.164819] LustreError: 357358:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12533.153038] LustreError: 356273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788863202 with bad export cookie 7576482217224926270 [12533.159964] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12533.166809] LustreError: 356273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12533.605751] Lustre: server umount lustre-MDT0001 complete [12542.481105] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [12545.120641] Key type lgssc unregistered [12545.417956] LNet: 359919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12545.436519] LNetError: 359919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12545.457597] LNet: Removed LNI 192.168.203.150@tcp [12546.453147] Key type .llcrypt unregistered [12546.457169] Key type ._llcrypt unregistered [12575.172914] Key type ._llcrypt registered [12575.176472] Key type .llcrypt registered [12575.285523] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [12576.163976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12576.183043] alg: No test for adler32 (adler32-zlib) [12577.291728] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [12577.573538] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [12579.264301] Key type lgssc registered [12580.297681] Lustre: Echo OBD driver; http://www.lustre.org/ [12587.742211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12587.752764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12589.131865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12593.183507] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12599.268961] LustreError: 361498:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12599.298694] LustreError: 361498:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12601.231507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12601.677521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12606.624482] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12613.884753] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12619.857625] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12626.668768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12626.982483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12632.712459] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12639.638481] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12651.490330] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12651.500958] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12651.515852] Lustre: Skipped 1 previous similar message [12651.537488] Lustre: Skipped 1 previous similar message [12652.362736] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.50@tcp (stopping) [12656.469997] Lustre: server umount lustre-MDT0001 complete [12656.611034] LustreError: 363948:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12656.633182] LustreError: 363948:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12661.729749] LustreError: 361498:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12661.742795] LustreError: 361498:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12666.208669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12666.336230] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12666.340820] Lustre: Found index 1 for lustre-MDT0001, updating log [12666.398127] Lustre: 361485:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12666.688748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12670.750184] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12671.998601] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12675.573480] LustreError: 365100:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12675.579950] LustreError: 365100:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12675.584898] LustreError: 361490:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12675.634591] LustreError: 365103:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.150@tcp: cfg command failed: rc = -17 [12675.646780] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [12675.646780] [12979.172199] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12979.172865] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12979.190352] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12979.200179] Lustre: Skipped 1 previous similar message [12982.596291] Lustre: server umount lustre-MDT0001 complete [12984.294602] LustreError: 361499:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12984.309382] LustreError: 361499:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12989.413977] LustreError: 366173:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12989.428689] LustreError: 366173:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12990.847844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12990.936074] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12990.944854] Lustre: Found index 1 for lustre-MDT0001, updating log [12990.978075] Lustre: 361486:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12991.320411] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12994.990155] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12996.586305] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12998.639935] LustreError: 374185:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12998.649193] LustreError: 374185:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12998.658269] LustreError: 361490:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12998.663061] LustreError: 361490:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12998.711472] LustreError: 374188:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [12998.719534] LustreError: 374188:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [12998.726744] LustreError: 374188:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.150@tcp: cfg command failed: rc = -17 [12998.735546] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.203.150@tcp [12998.735546] [13302.599115] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.50@tcp (stopping) [13302.611135] Lustre: Skipped 1 previous similar message [13303.781717] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13303.786357] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13303.805206] Lustre: Skipped 2 previous similar messages [13305.241910] Lustre: server umount lustre-MDT0001 complete [13307.708708] LustreError: 361493:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13307.754016] LustreError: 361493:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13316.202576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13316.346945] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13316.353174] Lustre: Found index 1 for lustre-MDT0001, updating log [13316.431415] Lustre: 361488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13316.729390] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13320.907387] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [13322.230239] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13324.275276] LustreError: 383147:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13324.279218] LustreError: 383147:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13324.298387] LustreError: 361490:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13324.307992] LustreError: 361490:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13324.384780] LustreError: 383150:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [13324.398764] LustreError: 383150:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.150@tcp: cfg command failed: rc = -114 [13324.404675] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [13324.404675] [13644.270465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13644.278417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13644.293158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13644.298705] Lustre: Skipped 2 previous similar messages [13647.726928] Lustre: server umount lustre-MDT0000 complete [13649.892553] LustreError: 361495:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13649.918401] LustreError: 361495:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [13651.175851] LustreError: 364685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788864320 with bad export cookie 2269758891896259246 [13651.186894] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13655.013154] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13655.027985] Lustre: Skipped 2 previous similar messages [13655.036032] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13655.041296] Lustre: Skipped 2 previous similar messages [13657.537518] Lustre: server umount lustre-MDT0001 complete [13661.167672] Lustre: server umount lustre-OST0000 complete [13671.839851] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [13681.931621] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [13716.636503] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 06:46:24 (1788864384) [13723.025174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13723.324204] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13723.343961] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13723.434404] Lustre: lustre-MDT0000: new disk, initializing [13723.542705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13723.564263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13728.191687] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [13737.980778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13738.074542] Lustre: 394076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13738.117326] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13738.120867] Lustre: Skipped 1 previous similar message [13738.177899] Lustre: lustre-MDT0001: new disk, initializing [13738.264225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13738.304883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13738.314085] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13742.220774] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [13751.638878] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13757.042678] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13763.524111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13763.772293] Lustre: lustre-OST0000: new disk, initializing [13763.776659] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13763.782288] Lustre: 396044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13763.875446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13769.890768] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [13773.913506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13773.919784] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13774.010057] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13778.843590] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13788.136949] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13788.146687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13788.175413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13788.184089] Lustre: Skipped 1 previous similar message [13792.438442] Lustre: server umount lustre-MDT0000 complete [13794.794812] LustreError: 394886:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13794.810403] LustreError: 394886:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [13796.194482] LustreError: 394871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788864465 with bad export cookie 2269758891896261199 [13796.198394] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13796.198927] LustreError: 394883:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [13796.224041] LustreError: 394871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [13796.643223] Lustre: server umount lustre-MDT0001 complete [13810.991439] Lustre: server umount lustre-OST0000 complete [13839.245411] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [13848.731408] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [13875.490505] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13578 sec ========= 06:49:03 (1788864543) [13876.780826] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:49:05 (1788864545) === [13879.587849] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:49:07 (1788864547) === [13921.964422] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [13925.141780] Key type lgssc unregistered [13925.367596] LNet: 402409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13925.378365] LNetError: 402409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13925.405608] LNet: Removed LNI 192.168.203.150@tcp [13926.273131] Key type .llcrypt unregistered [13926.276299] Key type ._llcrypt unregistered