[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 468181351 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002407] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.005005] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.009017] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011104] LSM: Security Framework initializing [ 0.012044] Yama: becoming mindful. [ 0.012751] SELinux: Initializing. [ 0.013095] *** VALIDATE selinux *** [ 0.022006] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027365] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030118] *** VALIDATE tmpfs *** [ 0.032416] *** VALIDATE proc *** [ 0.034017] *** VALIDATE cgroup *** [ 0.035011] *** VALIDATE cgroup2 *** [ 0.036273] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037158] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040002] Spectre V2 : User space: Vulnerable [ 0.041008] Speculative Store Bypass: Vulnerable [ 0.043922] debug: unmapping init [mem 0xffffffffbaa59000-0xffffffffbaa60fff] [ 0.046000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046726] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047022] ... version: 2 [ 0.047848] ... bit width: 48 [ 0.048017] ... generic registers: 4 [ 0.048914] ... value mask: 0000ffffffffffff [ 0.049016] ... max period: 00007fffffffffff [ 0.050014] ... fixed-purpose events: 3 [ 0.051013] ... event mask: 000000070000000f [ 0.052282] rcu: Hierarchical SRCU implementation. [ 0.054433] smp: Bringing up secondary CPUs ... [ 0.055571] x86: Booting SMP configuration: [ 0.056020] .... node #0, CPUs: #1 #2 #3 [ 0.059288] smp: Brought up 1 node, 4 CPUs [ 0.060954] smpboot: Max logical packages: 1 [ 0.061014] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.134959] node 0 deferred pages initialised in 71ms [ 0.137095] devtmpfs: initialized [ 0.138337] x86/mm: Memory block size: 128MB [ 0.140712] gcov: version magic: 0x41383552 [ 0.143278] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.146106] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.148292] pinctrl core: initialized pinctrl subsystem [ 0.150157] [ 0.150706] ************************************************************* [ 0.153013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.154006] ** ** [ 0.155009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.156006] ** ** [ 0.158009] ** This means that this kernel is built to expose internal ** [ 0.160008] ** IOMMU data structures, which may compromise security on ** [ 0.161006] ** your system. ** [ 0.163011] ** ** [ 0.165010] ** If you see this message and you are not debugging the ** [ 0.167010] ** kernel, report this immediately to your vendor! ** [ 0.168000] ** ** [ 0.169011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171010] ************************************************************* [ 0.173506] NET: Registered protocol family 16 [ 0.174342] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.177054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.179049] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.182134] cpuidle: using governor menu [ 0.183754] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.186371] PCI: Using configuration type 1 for base access [ 0.189398] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.198226] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.201031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.204178] cryptd: max_cpu_qlen set to 1000 [ 0.207862] ACPI: Added _OSI(Module Device) [ 0.209013] ACPI: Added _OSI(Processor Device) [ 0.211014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.213013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.218283] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.224350] ACPI: Interpreter enabled [ 0.225000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.227016] ACPI: Using IOAPIC for interrupt routing [ 0.229140] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.233443] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.244305] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.246056] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.248018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.251084] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.257369] acpiphp: Slot [2] registered [ 0.259160] acpiphp: Slot [5] registered [ 0.260144] acpiphp: Slot [6] registered [ 0.262151] acpiphp: Slot [7] registered [ 0.263111] acpiphp: Slot [8] registered [ 0.265102] acpiphp: Slot [9] registered [ 0.266105] acpiphp: Slot [10] registered [ 0.267127] acpiphp: Slot [3] registered [ 0.269102] acpiphp: Slot [4] registered [ 0.270064] acpiphp: Slot [11] registered [ 0.271081] acpiphp: Slot [12] registered [ 0.272114] acpiphp: Slot [13] registered [ 0.274092] acpiphp: Slot [14] registered [ 0.275075] acpiphp: Slot [15] registered [ 0.277116] acpiphp: Slot [16] registered [ 0.278085] acpiphp: Slot [17] registered [ 0.279111] acpiphp: Slot [18] registered [ 0.281093] acpiphp: Slot [19] registered [ 0.282094] acpiphp: Slot [20] registered [ 0.284090] acpiphp: Slot [21] registered [ 0.285119] acpiphp: Slot [22] registered [ 0.286099] acpiphp: Slot [23] registered [ 0.288107] acpiphp: Slot [24] registered [ 0.289098] acpiphp: Slot [25] registered [ 0.290092] acpiphp: Slot [26] registered [ 0.291102] acpiphp: Slot [27] registered [ 0.293121] acpiphp: Slot [28] registered [ 0.295133] acpiphp: Slot [29] registered [ 0.296104] acpiphp: Slot [30] registered [ 0.297073] acpiphp: Slot [31] registered [ 0.298016] PCI host bridge to bus 0000:00 [ 0.300018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.302021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.304023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.307034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.309021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.311044] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.313217] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.317292] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.320543] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.332025] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.337586] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.340030] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.342031] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.345029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.348480] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.351923] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.355046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.358808] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.364019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.376016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.382019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.388886] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.398025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.408022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.432036] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.443566] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.450021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.460018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.474023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.483033] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.495014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.499787] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.510020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.519944] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.533022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.550025] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.584022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.599794] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.609020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.615016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.632013] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.643737] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.653015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.658019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.677020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.687540] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.692466] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.695431] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.697422] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.700269] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.705173] iommu: Default domain type: Passthrough [ 0.707694] SCSI subsystem initialized [ 0.709167] ACPI: bus type USB registered [ 0.711169] usbcore: registered new interface driver usbfs [ 0.713119] usbcore: registered new interface driver hub [ 0.715138] usbcore: registered new device driver usb [ 0.718242] pps_core: LinuxPPS API ver. 1 registered [ 0.720014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.723090] PTP clock support registered [ 0.725176] EDAC MC: Ver: 3.0.0 [ 0.728158] PCI: Using ACPI for IRQ routing [ 0.731008] NetLabel: Initializing [ 0.733013] NetLabel: domain hash size = 128 [ 0.734013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.736102] NetLabel: unlabeled traffic allowed by default [ 0.739056] vgaarb: loaded [ 0.740324] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.742020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.750478] clocksource: Switched to clocksource kvm-clock [ 0.863798] VFS: Disk quotas dquot_6.6.0 [ 0.865512] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.867935] *** VALIDATE ramfs *** [ 0.869230] *** VALIDATE hugetlbfs *** [ 0.871412] pnp: PnP ACPI init [ 0.873908] pnp: PnP ACPI: found 6 devices [ 0.893149] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.896593] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.898824] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.901284] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.903951] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.906399] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.909486] NET: Registered protocol family 2 [ 0.912776] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.918795] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.923238] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.929962] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.933651] TCP: Hash tables configured (established 65536 bind 65536) [ 0.937181] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.940824] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.944254] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.947662] NET: Registered protocol family 1 [ 0.950598] RPC: Registered named UNIX socket transport module. [ 0.953035] RPC: Registered udp transport module. [ 0.955024] RPC: Registered tcp transport module. [ 0.957040] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.959693] NET: Registered protocol family 44 [ 0.961680] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.963894] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.966170] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.968853] PCI: CLS 0 bytes, default 64 [ 0.970778] Unpacking initramfs... [ 2.400661] debug: unmapping init [mem 0xffff8cd37cc54000-0xffff8cd37ffbffff] [ 2.407043] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.409191] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.413282] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.943576] Initialise system trusted keyrings [ 2.945381] Key type blacklist registered [ 2.950161] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.959237] zbud: loaded [ 2.962063] *** VALIDATE nfs *** [ 2.963000] *** VALIDATE nfs4 *** [ 2.964729] pstore: using deflate compression [ 2.969343] Platform Keyring initialized [ 3.078532] NET: Registered protocol family 38 [ 3.080460] Key type asymmetric registered [ 3.082580] Asymmetric key parser 'x509' registered [ 3.084577] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.088412] io scheduler mq-deadline registered [ 3.090064] io scheduler kyber registered [ 3.091708] io scheduler bfq registered [ 3.094155] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.097179] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.099702] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.101799] ACPI: Power Button [PWRF] [ 3.106978] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.113851] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.137470] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.146820] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.169775] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.200419] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.230457] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.235854] Non-volatile memory driver v1.3 [ 3.237768] Linux agpgart interface v0.103 [ 3.267667] virtio_blk virtio1: [vda] 145224 512-byte logical blocks (74.4 MB/70.9 MiB) [ 3.270345] vda: detected capacity change from 0 to 74354688 [ 3.286199] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.289442] vdb: detected capacity change from 0 to 1073741824 [ 3.305927] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.309295] vdc: detected capacity change from 0 to 2621440000 [ 3.325906] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.328709] vdd: detected capacity change from 0 to 2621440000 [ 3.344796] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.347602] vde: detected capacity change from 0 to 4294967296 [ 3.362101] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.365206] vdf: detected capacity change from 0 to 4294967296 [ 3.372798] libphy: Fixed MDIO Bus: probed [ 3.379762] usbcore: registered new interface driver usbserial_generic [ 3.382216] usbserial: USB Serial support registered for generic [ 3.384034] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.387702] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.389072] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.391158] mousedev: PS/2 mouse device common for all mice [ 3.394350] rtc_cmos 00:05: RTC can wake from S4 [ 3.396869] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.397634] rtc_cmos 00:05: registered as rtc0 [ 3.402273] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.404567] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.405743] intel_pstate: CPU model not supported [ 3.410916] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.415345] hid: raw HID events driver (C) Jiri Kosina [ 3.417242] usbcore: registered new interface driver usbhid [ 3.419363] usbhid: USB HID core driver [ 3.421138] drop_monitor: Initializing network drop monitor service [ 3.423575] Initializing XFRM netlink socket [ 3.425708] NET: Registered protocol family 10 [ 3.429181] Segment Routing with IPv6 [ 3.430630] NET: Registered protocol family 17 [ 3.433701] mpls_gso: MPLS GSO support [ 3.439205] RAS: Correctable Errors collector initialized. [ 3.441214] AVX version of gcm_enc/dec engaged. [ 3.442666] AES CTR mode by8 optimization enabled [ 3.534588] sched_clock: Marking stable (3534556358, 0)->(4426128162, -891571804) [ 3.538503] registered taskstats version 1 [ 3.542429] Loading compiled-in X.509 certificates [ 3.544445] zswap: loaded using pool lzo/zbud [ 3.571107] Key type big_key registered [ 3.583514] Key type encrypted registered [ 3.585074] ima: No TPM chip found, activating TPM-bypass! [ 3.586931] ima: Allocated hash algorithm: sha1 [ 3.588517] ima: No architecture policies found [ 3.590082] evm: Initialising EVM extended attributes: [ 3.591793] evm: security.selinux [ 3.592903] evm: security.ima [ 3.593884] evm: security.capability [ 3.595090] evm: HMAC attrs: 0x1 [ 3.597390] rtc_cmos 00:05: setting system clock to 2026-06-23 18:56:59 UTC (1782241019) [ 3.603893] debug: unmapping init [mem 0xffffffffbba03000-0xffffffffbbbfffff] [ 3.606743] debug: unmapping init [mem 0xffffffffba782000-0xffffffffbaa58fff] [ 3.615092] Write protecting the kernel read-only data: 28672k [ 3.618432] debug: unmapping init [mem 0xffffffffb8e03000-0xffffffffb8ffffff] [ 3.621019] debug: unmapping init [mem 0xffffffffb9714000-0xffffffffb97fffff] [ 3.655425] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.663425] systemd[1]: Detected virtualization kvm. [ 3.665207] systemd[1]: Detected architecture x86-64. [ 3.666918] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.693698] systemd[1]: No hostname configured. [ 3.695319] systemd[1]: Set hostname to . [ 3.697353] random: systemd: uninitialized urandom read (16 bytes read) [ 3.699664] systemd[1]: Initializing machine ID from random generator. [ 3.741476] random: ln: uninitialized urandom read (6 bytes read) [ 3.843212] random: systemd: uninitialized urandom read (16 bytes read) [ 3.846488] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.853855] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.860365] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.542538] device-mapper: uevent: version 1.0.3 [ 4.545060] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.229750] random: fast init done [ 5.234073] virtio_net virtio0 ens2: renamed from eth0 [ 5.283781] scsi host0: ata_piix [ 5.289738] scsi host1: ata_piix [ 5.292143] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.294831] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.918191] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.101909] random: crng init done [ 10.103417] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.482406] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.592662] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.876520] SELinux: Disabled at runtime. [ 11.937600] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.947593] systemd[1]: Detected virtualization kvm. [ 11.949416] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.490692] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.495085] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.499679] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.504540] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.508591] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.516377] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.524124] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Local Encrypted Volumes. [ 12.605765] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 12.975306] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.280094] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.296722] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.406638] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.419736] EDAC sbridge: Ver: 1.1.2 [ 15.236816] Key type dns_resolver registered [ 15.551217] NFS: Registering the id_resolver key type [ 15.553050] Key type id_resolver registered [ 15.554341] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg429-server login: [ 62.442592] libcfs: loading out-of-tree module taints kernel. [ 62.489590] Key type ._llcrypt registered [ 62.492658] Key type .llcrypt registered [ 62.590151] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 80.949619] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 82.356625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 82.374275] alg: No test for adler32 (adler32-zlib) [ 83.708363] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 84.587319] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 86.375176] Key type lgssc registered [ 87.746911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 105.036391] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 145.619996] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 153.502120] hrtimer: interrupt took 9109363 ns [ 158.330862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 158.352050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 159.568342] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 159.628240] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 159.710333] Lustre: lustre-MDT0000: new disk, initializing [ 159.768344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 159.786117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 164.331357] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.535076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 177.603668] Lustre: 6512:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 177.626471] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 177.632095] Lustre: Skipped 1 previous similar message [ 177.704284] Lustre: lustre-MDT0001: new disk, initializing [ 177.764334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 177.803285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 177.815905] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 181.688842] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 186.319422] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 196.286558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 196.485361] Lustre: lustre-OST0000: new disk, initializing [ 196.488834] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 196.494727] Lustre: 8450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 196.572812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.558080] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.910319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 205.921590] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 205.988478] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 216.809723] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 216.956621] Lustre: lustre-OST0001: new disk, initializing [ 216.960682] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 216.965710] Lustre: 9523:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 217.041429] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 223.817809] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.840193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 225.862353] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 225.918111] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 236.566887] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 244.108398] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 250.577819] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing check_logdir /tmp/testlogs/ [ 256.208650] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing yml_node [ 261.726615] Lustre: DEBUG MARKER: Client: 2.17.54.83 [ 264.606768] Lustre: DEBUG MARKER: MDS: 2.17.54.83 [ 267.109602] Lustre: DEBUG MARKER: OSS: 2.17.54.83 [ 269.140900] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 23 15:01:23 EDT 2026 [ 287.064499] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 289.049962] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 305.645339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 305.648850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 305.671923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 307.684053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 307.693384] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 307.704776] Lustre: Skipped 2 previous similar messages [ 311.041813] Lustre: server umount lustre-MDT0000 complete [ 315.120692] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782241331 with bad export cookie 6929821902818468927 [ 315.138129] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 315.139049] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 317.920216] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 317.922448] LustreError: 8179:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 317.952719] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 317.994593] LustreError: 8179:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 318.024030] Lustre: Skipped 4 previous similar messages [ 320.929852] LustreError: 8179:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 320.930576] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 320.970205] LustreError: 8179:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 320.994390] Lustre: Skipped 1 previous similar message [ 321.618753] Lustre: server umount lustre-MDT0001 complete [ 326.000110] Lustre: server umount lustre-OST0000 complete [ 330.164973] Lustre: server umount lustre-OST0001 complete [ 337.007625] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 346.958624] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 380.661289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 380.865865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 380.915062] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 381.052627] Lustre: lustre-MDT0000: new disk, initializing [ 381.212736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 381.239035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 385.650193] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 396.132580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 396.264529] Lustre: 15088:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 396.310231] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 396.313423] Lustre: Skipped 1 previous similar message [ 396.367288] Lustre: lustre-MDT0001: new disk, initializing [ 396.462306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 396.499414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 396.515136] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 400.970319] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 410.118364] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 416.376550] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 423.320673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 423.637468] Lustre: lustre-OST0000: new disk, initializing [ 423.640586] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 423.653389] Lustre: 17055:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 423.772912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 425.337829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 425.342866] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 425.504032] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 430.018018] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 440.529809] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 444.560176] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 444.702456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 449.717719] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 449.994384] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 456.170609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 456.171912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 456.191081] Lustre: Skipped 1 previous similar message [ 456.195733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 458.471363] Lustre: server umount lustre-OST0000 complete [ 463.335092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 463.343898] LustreError: Skipped 1 previous similar message [ 463.351113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 463.368686] Lustre: Skipped 1 previous similar message [ 466.412792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 466.417558] Lustre: Skipped 3 previous similar messages [ 468.528968] Lustre: server umount lustre-MDT0000 complete [ 471.521045] LustreError: 15963:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 471.547696] LustreError: 15963:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 472.249212] LustreError: 15079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782241488 with bad export cookie 6929821902818470460 [ 472.252767] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 472.255415] LustreError: 15079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 472.716952] Lustre: server umount lustre-MDT0001 complete [ 478.237285] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 15:04:52 (1782241492) [ 484.807800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 485.168407] LustreError: 19323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 485.223292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 489.079594] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 490.466320] LustreError: 19324:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 498.497781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 503.693669] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 512.562694] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 518.921990] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 528.486799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 538.146665] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 546.667480] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 550.085500] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 555.489228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 555.493317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 555.499433] Lustre: Skipped 1 previous similar message [ 555.502444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 560.786418] Lustre: server umount lustre-MDT0000 complete [ 564.205900] LustreError: 20327:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 564.246872] LustreError: 20327:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 565.703514] LustreError: 21854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782241581 with bad export cookie 6929821902818471342 [ 565.706327] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 565.709136] LustreError: 21854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 566.131977] Lustre: server umount lustre-MDT0001 complete [ 581.320401] Lustre: server umount lustre-OST0000 complete [ 609.575588] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 621.098894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 621.469399] LustreError: 23729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 621.564165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 621.574165] Lustre: Skipped 2 previous similar messages [ 626.554664] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 636.573712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 636.900923] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 636.908200] Lustre: Skipped 3 previous similar messages [ 642.406540] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 645.687482] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 653.114211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 661.159878] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 670.242839] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 670.394691] Lustre: lustre-OST0001: new disk, initializing [ 670.398041] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 670.404391] Lustre: 26045:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 672.127672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 672.156199] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 672.280205] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 676.746892] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 689.299447] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 693.811664] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 708.073676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 708.081446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 708.094478] Lustre: Skipped 5 previous similar messages [ 708.110278] Lustre: Skipped 3 previous similar messages [ 713.019906] Lustre: server umount lustre-MDT0000 complete [ 713.193791] LustreError: 23729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 713.230145] LustreError: 23729:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 716.579361] LustreError: 23709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782241732 with bad export cookie 6929821902818472189 [ 716.582147] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 716.589534] LustreError: 23709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 717.000616] Lustre: server umount lustre-MDT0001 complete [ 730.296906] Lustre: server umount lustre-OST0000 complete [ 744.305818] Lustre: server umount lustre-OST0001 complete [ 763.398338] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 777.369597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 778.113620] LustreError: 29067:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 778.204148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 778.215874] Lustre: Skipped 3 previous similar messages [ 784.024584] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 794.021251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 800.018656] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 803.583731] Lustre: 30215:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 810.318807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 816.851282] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 826.440905] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 833.199931] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 841.354954] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 844.963394] Lustre: 32092:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 852.463027] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 852.466963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 852.476704] Lustre: Skipped 2 previous similar messages [ 852.492637] Lustre: Skipped 3 previous similar messages [ 857.570531] Lustre: server umount lustre-MDT0000 complete [ 861.264978] LustreError: 32094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782241877 with bad export cookie 6929821902818473484 [ 861.270392] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 861.286257] LustreError: 32094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 861.593966] Lustre: server umount lustre-MDT0001 complete [ 875.570235] Lustre: server umount lustre-OST0000 complete [ 878.559175] Lustre: 3646:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782241878/real 1782241878] req@ffff8cd3efe1fb80 x1868815243973248/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782241894 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 880.375763] Lustre: server umount lustre-OST0001 complete [ 891.479585] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 15:11:45 (1782241905) [ 922.856572] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 933.389257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 933.768645] LustreError: 34896:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 933.787745] LustreError: 34896:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 938.159483] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 947.444272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 952.212811] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 955.280426] Lustre: 36043:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 961.619857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 969.348434] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 978.212900] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 984.947851] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 992.507949] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 996.685812] Lustre: 37926:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1004.008313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1004.021592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1004.047398] Lustre: Skipped 2 previous similar messages [ 1008.665392] Lustre: server umount lustre-MDT0000 complete [ 1012.312877] LustreError: 36045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782242028 with bad export cookie 6929821902818474590 [ 1012.326575] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1012.328613] LustreError: 36045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1012.627300] Lustre: server umount lustre-MDT0001 complete [ 1027.542989] Lustre: server umount lustre-OST0000 complete [ 1041.662450] Lustre: server umount lustre-OST0001 complete [ 1051.663595] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 15:14:26 (1782242066) [ 1072.527638] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1084.324304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1084.872278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1084.882625] Lustre: Skipped 7 previous similar messages [ 1089.709728] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1097.962452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1102.992977] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1106.309054] Lustre: 41217:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1113.954392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1120.551677] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1129.757195] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1137.726366] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1146.773803] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1152.161680] Lustre: 43096:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1153.017506] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1153.025647] Lustre: Skipped 1 previous similar message [ 1156.048851] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.29@tcp (stopping) [ 1156.065911] Lustre: Skipped 5 previous similar messages [ 1159.651533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1159.664581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1159.690681] Lustre: Skipped 3 previous similar messages [ 1162.258791] Lustre: server umount lustre-MDT0000 complete [ 1166.333956] LustreError: 40047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782242182 with bad export cookie 6929821902818475689 [ 1166.341631] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1166.365598] LustreError: 40047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1172.780283] Lustre: server umount lustre-MDT0001 complete [ 1182.443470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1191.394777] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x602bac4e0a378317 [ 1191.408407] Lustre: MGC192.168.204.129@tcp: Connection restored to 0@lo (at 0@lo) [ 1191.710367] LustreError: 41581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1191.738644] LustreError: 41581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 23 previous similar messages [ 1196.211458] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1202.091434] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1203.508543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1208.211940] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1209.334979] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1209.342114] Lustre: Skipped 1 previous similar message [ 1216.264397] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1218.367030] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1223.318308] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1225.132849] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1234.401199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1236.502865] Lustre: server umount lustre-MDT0000 complete [ 1240.272745] LustreError: 41219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782242256 with bad export cookie 6929821902818476823 [ 1240.297360] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1240.596533] Lustre: server umount lustre-MDT0001 complete [ 1254.472838] Lustre: server umount lustre-OST0000 complete [ 1268.930825] Lustre: server umount lustre-OST0001 complete [ 1277.268313] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 15:18:11 (1782242291) [ 1291.812830] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1300.603486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1305.105546] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1312.859850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1317.244161] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1319.728186] Lustre: 48856:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1325.630979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1332.148333] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1339.935535] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1346.370315] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1354.131319] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1359.359266] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1373.935095] Lustre: server umount lustre-MDT0000 complete [ 1374.689537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1377.493967] LustreError: 47685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782242393 with bad export cookie 6929821902818477614 [ 1377.498255] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1377.506991] LustreError: 47685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1377.806376] Lustre: server umount lustre-MDT0001 complete [ 1392.012890] Lustre: server umount lustre-OST0000 complete [ 1406.030740] Lustre: server umount lustre-OST0001 complete [ 1416.041655] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 15:20:30 (1782242430) [ 1418.664085] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1420.791995] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 15:20:35 (1782242435) [ 1430.665930] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 1440.201443] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1461.874664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1462.149034] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1462.237330] Lustre: lustre-MDT0000: new disk, initializing [ 1462.344945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1466.900649] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1475.712765] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1482.301048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1482.557340] Lustre: lustre-OST0000: new disk, initializing [ 1482.562120] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1482.576085] Lustre: Skipped 1 previous similar message [ 1482.580634] Lustre: 54932:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1484.349937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1484.368797] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1484.401200] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1488.100360] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1497.550880] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1500.181858] Lustre: DEBUG MARKER: create test files [ 1530.342096] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1530.342723] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1530.364724] Lustre: Skipped 13 previous similar messages [ 1536.225913] Lustre: server umount lustre-OST0000 complete [ 1539.945388] Lustre: server umount lustre-MDT0000 complete [ 1551.702928] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1554.823858] Key type lgssc unregistered [ 1555.144772] LNet: 56559:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1555.152451] LNetError: 56559:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1555.167785] LNet: Removed LNI 192.168.204.129@tcp [ 1556.238216] Key type .llcrypt unregistered [ 1556.241428] Key type ._llcrypt unregistered [ 1584.190635] Key type ._llcrypt registered [ 1584.194275] Key type .llcrypt registered [ 1584.312891] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1585.395437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1585.421901] alg: No test for adler32 (adler32-zlib) [ 1586.489768] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 1586.711936] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1588.375168] Key type lgssc registered [ 1589.398702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1596.496635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1596.520839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1597.900726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1602.395736] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1609.471261] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1616.551400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1617.070682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1622.196339] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 1622.286426] Lustre: 59292:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1622.291974] Lustre: 59292:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1622.296532] Lustre: 59292:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1622.300834] Lustre: 59292:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1622.305307] Lustre: 59292:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1622.311626] Lustre: 59292:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1622.609501] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1629.701741] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1634.786964] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1640.388579] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1673.700707] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1673.715124] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1673.731093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1678.154583] Lustre: server umount lustre-OST0000 complete [ 1682.747196] Lustre: server umount lustre-MDT0000 complete [ 1697.224654] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1700.697534] Key type lgssc unregistered [ 1700.997909] LNet: 60927:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1701.004904] LNetError: 60927:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.057440] LNet: Removed LNI 192.168.204.129@tcp [ 1703.171616] Key type .llcrypt unregistered [ 1703.173995] Key type ._llcrypt unregistered [ 1734.460557] Key type ._llcrypt registered [ 1734.463213] Key type .llcrypt registered [ 1734.586342] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1736.051407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1736.085856] alg: No test for adler32 (adler32-zlib) [ 1737.149742] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 1737.365507] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1739.047173] Key type lgssc registered [ 1740.293664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1749.466678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1749.501464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1751.047173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1755.485257] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1762.642611] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1769.136219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1769.446966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1775.516748] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1775.592189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1775.761924] Lustre: 63760:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1775.782508] Lustre: 63760:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1775.797974] Lustre: 63760:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1775.811466] Lustre: 63760:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1775.827060] Lustre: 63760:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1775.839555] Lustre: 63760:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1783.448344] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1788.734911] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1801.186057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1801.190800] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1801.199744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1804.754479] Lustre: server umount lustre-OST0000 complete [ 1809.333873] Lustre: server umount lustre-MDT0000 complete [ 1821.087695] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1823.820399] Key type lgssc unregistered [ 1824.201803] LNet: 65315:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1824.211181] LNetError: 65315:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1824.238565] LNet: Removed LNI 192.168.204.129@tcp [ 1825.130598] Key type .llcrypt unregistered [ 1825.138920] Key type ._llcrypt unregistered [ 1843.600590] Key type ._llcrypt registered [ 1843.602275] Key type .llcrypt registered [ 1843.725570] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 1863.466159] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1864.721816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1864.842602] alg: No test for adler32 (adler32-zlib) [ 1865.871969] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 1866.122539] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1867.823239] Key type lgssc registered [ 1869.327828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1906.031790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1906.061778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1907.317287] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1907.365277] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1907.503644] Lustre: lustre-MDT0000: new disk, initializing [ 1907.611023] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1907.634326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1912.483418] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1924.583790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1924.683525] Lustre: 68469:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1924.750980] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1924.754951] Lustre: Skipped 1 previous similar message [ 1924.830515] Lustre: lustre-MDT0001: new disk, initializing [ 1924.980594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1925.053812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1925.078301] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1930.439575] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1940.024066] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1945.908544] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1953.426535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1953.878474] Lustre: lustre-OST0000: new disk, initializing [ 1953.897948] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1953.907654] Lustre: 70440:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1954.068503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1960.229839] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1963.110768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1963.133278] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1963.288556] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1970.327396] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1975.166081] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1975.379051] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1980.341890] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1980.526286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1983.970351] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1983.980080] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.994709] LustreError: Skipped 1 previous similar message [ 1983.997664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1984.047439] Lustre: Skipped 1 previous similar message [ 1987.118910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1987.124475] Lustre: Skipped 2 previous similar messages [ 1988.504208] Lustre: server umount lustre-OST0000 complete [ 1992.161301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1992.164964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1992.169251] Lustre: Skipped 1 previous similar message [ 1992.183654] Lustre: Skipped 1 previous similar message [ 1997.283059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1997.291887] Lustre: Skipped 1 previous similar message [ 1998.602920] Lustre: server umount lustre-MDT0000 complete [ 2002.661613] LustreError: 68461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782243018 with bad export cookie 11258093432714229449 [ 2002.673314] LustreError: 68461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2002.677859] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2003.032706] Lustre: server umount lustre-MDT0001 complete [ 2012.606279] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 15:30:26 (1782243026) [ 2023.762159] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2062.539504] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 2073.328751] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2111.078423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2111.406527] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2111.440746] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2111.549446] Lustre: lustre-MDT0000: new disk, initializing [ 2111.651344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2111.682730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2116.313144] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2127.800812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2127.974231] Lustre: 76459:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2128.062302] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2128.072950] Lustre: Skipped 1 previous similar message [ 2128.186600] Lustre: lustre-MDT0001: new disk, initializing [ 2128.326838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2128.371690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2128.384279] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2133.567357] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2143.851524] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2150.982596] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2159.661756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2160.120300] Lustre: lustre-OST0000: new disk, initializing [ 2160.134476] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2160.145060] Lustre: 78426:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2160.272839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2166.324498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2166.342685] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2166.479325] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2167.577658] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2177.597933] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2182.886714] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2183.129184] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2187.988583] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2188.201948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2192.364922] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2192.368519] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2192.374204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2192.397020] Lustre: Skipped 1 previous similar message [ 2196.562665] Lustre: server umount lustre-OST0000 complete [ 2202.593744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2202.600530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2202.612198] Lustre: Skipped 1 previous similar message [ 2202.621775] Lustre: Skipped 1 previous similar message [ 2207.300708] Lustre: server umount lustre-MDT0000 complete [ 2207.714328] LustreError: 76471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2207.734846] LustreError: 76471:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2211.915635] LustreError: 79768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782243227 with bad export cookie 11258093432714230331 [ 2211.920153] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2211.934860] LustreError: 79768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2212.685614] Lustre: server umount lustre-MDT0001 complete [ 2222.940813] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 15:33:57 (1782243237) [ 2230.323477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2230.783162] LustreError: 80842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2230.930308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2236.049974] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2236.384695] LustreError: 80843:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2240.484081] LustreError: 80842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2244.622971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2249.528764] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2259.023516] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2266.790508] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2274.110732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2274.560062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2274.573604] Lustre: Skipped 1 previous similar message [ 2281.158930] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2289.227497] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2292.873460] Lustre: 83517:0:(genops.c:1795:obd_export_evict_by_uuid()) MGS: evicting 10d5e29a-35d5-42cc-a1fa-0dab01386d93 at adminstrative request [ 2295.269749] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2295.281459] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2305.513935] Lustre: MGS: Client 10d5e29a-35d5-42cc-a1fa-0dab01386d93 (at 0@lo) reconnecting [ 2305.523695] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x9c3cc8544116fdad to 0x9c3cc8544117004d [ 2305.542828] Lustre: MGC192.168.204.129@tcp: Connection restored to 0@lo (at 0@lo) [ 2329.995159] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2330.179893] Lustre: lustre-OST0001: new disk, initializing [ 2330.183441] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2330.189591] Lustre: 83908:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2335.820637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2335.841595] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2335.948455] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2336.673329] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2346.821340] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2351.596657] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2351.603737] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2351.614884] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2351.628220] LustreError: Skipped 1 previous similar message [ 2351.643100] Lustre: Skipped 1 previous similar message [ 2364.384414] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2364.476057] Lustre: server umount lustre-OST0001 complete [ 2366.957411] LustreError: 84212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.970574] LustreError: 84212:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2372.078311] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2372.083795] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2372.090815] Lustre: Skipped 2 previous similar messages [ 2374.422604] Lustre: server umount lustre-OST0000 complete [ 2378.209312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2384.151083] Lustre: server umount lustre-MDT0000 complete [ 2387.427708] LustreError: 80839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2387.447629] LustreError: 80839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2387.975325] LustreError: 80822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782243403 with bad export cookie 11258093432714231885 [ 2387.979591] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2387.986319] LustreError: 80822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2388.410320] Lustre: server umount lustre-MDT0001 complete [ 2397.067353] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 15:36:51 (1782243411) [ 2398.870878] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2400.410755] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 15:36:55 (1782243415) [ 2401.937792] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2403.994397] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 15:36:58 (1782243418) [ 2405.748422] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2407.372215] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 15:37:01 (1782243421) [ 2408.934398] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2410.542424] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 15:37:05 (1782243425) [ 2411.993489] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2417.365024] loop: module loaded [ 2417.382763] loop0: detected capacity change from 0 to 4294967296 [ 2417.431705] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2419.046476] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2422.732864] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2427.362109] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2458.528407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2458.583731] LustreError: 88036:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2458.591355] LustreError: 88036:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2458.598468] LustreError: 88036:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2458.605629] LustreError: 88036:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2479.742683] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 2491.498140] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2529.885378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2530.119176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2530.163759] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2530.256915] Lustre: lustre-MDT0000: new disk, initializing [ 2530.334375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2530.341263] Lustre: Skipped 1 previous similar message [ 2530.350771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2534.161237] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2544.053535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2544.189169] Lustre: 90906:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2544.418912] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2548.432626] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2558.469145] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2564.137130] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2571.072286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2571.393763] Lustre: 92877:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2577.012394] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2578.010231] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2578.094656] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2586.188401] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2590.888922] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2591.118387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2595.292878] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2595.527926] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2598.883612] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2598.891705] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2598.901077] LustreError: Skipped 1 previous similar message [ 2598.904098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2598.904105] Lustre: Skipped 9 previous similar messages [ 2598.941423] Lustre: Skipped 3 previous similar messages [ 2603.257967] Lustre: server umount lustre-OST0000 complete [ 2613.650244] Lustre: server umount lustre-MDT0000 complete [ 2614.240111] LustreError: 90914:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2614.265796] LustreError: 90914:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2617.547625] LustreError: 90900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782243633 with bad export cookie 11258093432714232501 [ 2617.554557] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2617.565405] LustreError: 90900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2618.103375] Lustre: server umount lustre-MDT0001 complete [ 2627.396717] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 15:40:41 (1782243641) [ 2634.761553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2640.079664] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2650.584078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2650.595059] LustreError: 95291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2650.619460] LustreError: 95291:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2656.468222] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2664.978539] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2671.665821] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2679.027690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2679.268863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2679.275248] Lustre: Skipped 4 previous similar messages [ 2685.795652] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2693.333815] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2698.324726] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2698.556796] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2703.038421] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2703.279981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2710.476101] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2710.664546] Lustre: lustre-OST0001: new disk, initializing [ 2710.669473] Lustre: Skipped 2 previous similar messages [ 2710.672852] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2710.680023] Lustre: Skipped 3 previous similar messages [ 2710.694417] Lustre: 98660:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2717.712448] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2719.827896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2719.836542] Lustre: Skipped 2 previous similar messages [ 2719.848503] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2719.932115] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2728.295475] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2733.648544] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2733.881106] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2739.643603] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2739.846765] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2761.075183] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2792.115729] Lustre: Failing over lustre-MDT0000 [ 2792.829773] Lustre: server umount lustre-MDT0000 complete [ 2793.427096] LustreError: 95284:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2793.445345] LustreError: 95284:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2794.476416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2794.488136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2794.506439] Lustre: Skipped 2 previous similar messages [ 2812.384538] Lustre: 66695:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782243812/real 1782243812] req@00000000c1c1a270 x1868817112518400/t0(0) o400->MGC192.168.204.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782243828 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2812.452616] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2818.801755] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2818.806881] LDISKFS-fs (dm-0): recovery complete [ 2818.832397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2822.627328] LustreError: 66691:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000729c6520 x1868817112525184/t0(0) o250->MGC192.168.204.129@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2823.012848] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2825.149595] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2825.152029] Lustre: 99296:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2828.264120] Lustre: 95291:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2828.285545] Lustre: 95291:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 1 previous similar message [ 2828.331557] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2828.424236] LustreError: 101511:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2828.752187] LustreError: 101511:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2829.103725] LustreError: 101511:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2829.126269] LustreError: 101511:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 1 previous similar message [ 2829.286397] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2829.463167] LustreError: 101511:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2829.485525] LustreError: 101511:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 1 previous similar message [ 2829.500709] Lustre: 101511:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2829.518845] Lustre: 101511:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 7 previous similar messages [ 2830.166314] LustreError: 101511:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2830.176669] LustreError: 101511:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 2 previous similar messages [ 2830.505381] LustreError: 101511:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2830.518620] LustreError: 101511:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 2 previous similar messages [ 2831.532058] Lustre: 101511:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2831.562614] Lustre: 101511:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 5 previous similar messages [ 2832.216798] LustreError: 101511:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2832.233414] LustreError: 101511:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 5 previous similar messages [ 2832.559172] LustreError: 101511:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2832.570753] LustreError: 101511:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 5 previous similar messages [ 2835.788351] Lustre: 101511:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2835.802559] Lustre: 101511:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 12 previous similar messages [ 2836.434382] LustreError: 101511:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2836.455801] LustreError: 101511:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 12 previous similar messages [ 2836.792074] LustreError: 101511:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2836.799166] LustreError: 101511:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 12 previous similar messages [ 2844.103642] Lustre: 101511:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2844.120671] Lustre: 101511:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 25 previous similar messages [ 2844.459742] LustreError: 101511:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2844.477682] LustreError: 101511:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 24 previous similar messages [ 2844.800191] LustreError: 101511:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2844.817804] LustreError: 101511:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 24 previous similar messages [ 2860.107545] Lustre: 101511:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2860.128374] Lustre: 101511:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 49 previous similar messages [ 2860.472772] LustreError: 101511:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2860.488349] LustreError: 101511:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 49 previous similar messages [ 2860.815136] LustreError: 101511:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2860.827218] LustreError: 101511:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 49 previous similar messages [ 2885.023118] Lustre: 101511:0:(ldlm_lib.c:2380:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2885.032145] Lustre: 101511:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2885.037260] Lustre: 101511:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 089ad016-bd89-4cea-a4a1-bc1e52a2a627@192.168.204.29@tcp [ 2885.044220] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2885.047986] Lustre: 101511:0:(ldlm_lib.c:1900:abort_req_replay_queue()) @@@ aborted: req@00000000d8c5fb03 x1868817092383360/t0(8589934773) o101->089ad016-bd89-4cea-a4a1-bc1e52a2a627@192.168.204.29@tcp:726/0 lens 592/0 e 0 to 0 dl 1782243911 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2885.067312] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2885.082433] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2885.089853] Lustre: lustre-MDT0000: Denying connection for new client 089ad016-bd89-4cea-a4a1-bc1e52a2a627 (at 192.168.204.29@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2885.130150] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2885.138091] Lustre: Skipped 2 previous similar messages [ 2885.159286] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2885.228388] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:46 to 0x2c0000401:545) [ 2885.234913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:78 to 0x280000401:577) [ 2910.204943] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2910.233545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2910.245358] Lustre: Skipped 4 previous similar messages [ 2910.285992] Lustre: Skipped 6 previous similar messages [ 2914.299404] Lustre: server umount lustre-OST0000 complete [ 2919.907621] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2925.422552] Lustre: server umount lustre-OST0001 complete [ 2936.914087] Lustre: server umount lustre-MDT0000 complete [ 2940.901818] LustreError: 96587:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2940.930422] LustreError: 96587:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 42 previous similar messages [ 2941.987080] LustreError: 95270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782243957 with bad export cookie 11258093432714283720 [ 2942.013440] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2942.633265] Lustre: server umount lustre-MDT0001 complete [ 2953.664527] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 15:46:07 (1782243967) [ 2962.668625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2962.674236] Lustre: *** cfs_fail_loc=197, val=0*** [ 2978.273134] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2983.391835] LustreError: 103492:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.129@tcp: failed processing log, type 4: rc = -110 [ 2988.545333] Lustre: *** cfs_fail_loc=197, val=0*** [ 2988.547333] Lustre: Skipped 65 previous similar messages [ 3009.120170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3009.127417] Lustre: Skipped 2 previous similar messages [ 3015.861806] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3024.846509] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3027.538793] Lustre: server umount lustre-OST0000 complete [ 3038.994552] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 15:47:32 (1782244052) [ 3062.078896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3068.357580] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3079.209572] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3084.161375] LustreError: 105147:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3084.168172] LustreError: 105147:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3084.182728] LustreError: 105147:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 3084.748900] Lustre: server umount lustre-MDT0000 complete [ 3123.288294] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 3135.162201] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 3179.389460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3179.599121] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3179.622981] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3179.686851] Lustre: lustre-MDT0000: new disk, initializing [ 3179.796261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3184.327473] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3195.304923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3195.427477] Lustre: 109333:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3195.667985] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3199.926189] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3211.021086] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3217.891538] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3225.513849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3225.881267] Lustre: 111303:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3233.411381] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3235.020456] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3244.759459] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3249.687475] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3249.931671] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3254.944394] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3255.188489] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3260.898670] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3260.906493] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3260.917648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3260.917655] Lustre: Skipped 7 previous similar messages [ 3260.943433] Lustre: Skipped 5 previous similar messages [ 3272.159333] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3272.462767] Lustre: server umount lustre-OST0000 complete [ 3283.257235] Lustre: server umount lustre-MDT0000 complete [ 3286.499778] LustreError: 109344:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3286.528087] LustreError: 109344:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3287.575368] LustreError: 109324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782244303 with bad export cookie 11258093432714294332 [ 3287.580027] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3287.587524] LustreError: 109324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3287.891916] Lustre: server umount lustre-MDT0001 complete [ 3297.265497] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 15:51:51 (1782244311) [ 3308.159596] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 3311.502780] Key type lgssc unregistered [ 3311.848044] LNet: 113794:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3311.861414] LNetError: 113794:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3311.877875] LNet: Removed LNI 192.168.204.129@tcp [ 3312.728420] Key type .llcrypt unregistered [ 3312.732895] Key type ._llcrypt unregistered [ 3332.440112] Key type ._llcrypt registered [ 3332.444382] Key type .llcrypt registered [ 3332.560428] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 3350.225473] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 3351.452657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3351.662226] alg: No test for adler32 (adler32-zlib) [ 3352.688221] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 3352.909038] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 3354.615233] Key type lgssc registered [ 3355.944060] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3404.215854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3404.281790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3405.620653] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3405.677814] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3405.775226] Lustre: lustre-MDT0000: new disk, initializing [ 3405.934134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3405.959493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3411.426946] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3423.421363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3423.810570] Lustre: lustre-OST0000: new disk, initializing [ 3423.818152] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3423.820954] Lustre: Skipped 1 previous similar message [ 3423.836671] Lustre: 117968:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3423.939580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3430.689106] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3432.581439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3432.599220] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3432.640714] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3445.596458] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3445.822545] Lustre: lustre-OST0001: new disk, initializing [ 3445.826224] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3445.833110] Lustre: 118831:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3445.910055] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3453.613026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3453.637183] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3453.680707] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3454.612954] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3468.769558] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3468.772894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3468.805199] Lustre: Skipped 1 previous similar message [ 3468.819465] Lustre: Skipped 1 previous similar message [ 3471.316779] Lustre: server umount lustre-MDT0000 complete [ 3478.299970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3482.946391] LustreError: 117154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782244498 with bad export cookie 6169102920092342030 [ 3482.948499] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3482.976602] LustreError: 117154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3483.114391] Lustre: server umount lustre-OST0000 complete [ 3487.224424] Lustre: server umount lustre-OST0001 complete [ 3497.046443] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 15:55:11 (1782244511) [ 3507.411440] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 3534.617975] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 15:55:48 (1782244548) [ 3579.887201] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 3590.940471] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 3636.283438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3636.615756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3636.660660] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3636.750930] Lustre: lustre-MDT0000: new disk, initializing [ 3636.908913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3636.964384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3642.524567] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3654.527221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3654.609343] Lustre: 125585:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3654.663345] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3654.669059] Lustre: Skipped 1 previous similar message [ 3654.736146] Lustre: lustre-MDT0001: new disk, initializing [ 3654.798466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3654.843224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3654.852214] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3659.733873] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3671.000309] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3678.175244] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3686.364994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3686.611534] Lustre: lustre-OST0000: new disk, initializing [ 3686.614694] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3686.620738] Lustre: 127553:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3686.704798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3692.661833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3692.668603] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3692.800532] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3694.057598] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3704.666299] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3709.890729] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3710.119211] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3715.235905] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3715.535132] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3718.630942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3718.638442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3718.654942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3718.666099] Lustre: Skipped 1 previous similar message [ 3723.878980] Lustre: server umount lustre-OST0000 complete [ 3728.868244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3728.876312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3728.883362] Lustre: Skipped 2 previous similar messages [ 3728.894742] Lustre: Skipped 1 previous similar message [ 3733.985336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3733.990349] Lustre: Skipped 1 previous similar message [ 3739.105386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3739.117657] Lustre: Skipped 1 previous similar message [ 3742.175222] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3742.510571] Lustre: server umount lustre-MDT0000 complete [ 3744.224941] LustreError: 126954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3744.241973] LustreError: 126954:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3746.823816] LustreError: 125576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782244762 with bad export cookie 6169102920092343255 [ 3746.831577] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3746.834754] LustreError: 125576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3747.151554] Lustre: server umount lustre-MDT0001 complete [ 3758.427510] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 15:59:32 (1782244772) [ 3766.198642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3766.731782] LustreError: 129966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3766.827822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3771.876100] LustreError: 129967:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3771.973978] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3776.997179] LustreError: 129966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3781.688768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3786.580172] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3795.308907] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3801.881474] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3809.374786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3809.756247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3809.761073] Lustre: Skipped 1 previous similar message [ 3816.409858] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3824.131922] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3832.293401] Lustre: *** cfs_fail_loc=159, val=0*** [ 3832.296312] Lustre: Skipped 1 previous similar message [ 3848.692870] Lustre: lustre-MDT0000: Client 15f8397b-efad-45d3-9cb3-764a09d8af66 (at 192.168.204.29@tcp) reconnecting [ 3850.606845] Lustre: *** cfs_fail_loc=159, val=0*** [ 3850.609854] Lustre: Skipped 4 previous similar messages [ 3867.134270] Lustre: lustre-MDT0000: Client 15f8397b-efad-45d3-9cb3-764a09d8af66 (at 192.168.204.29@tcp) reconnecting [ 3877.346080] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3877.363935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3877.365054] LustreError: Skipped 1 previous similar message [ 3877.370171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3877.410755] Lustre: Skipped 1 previous similar message [ 3879.740657] Lustre: server umount lustre-OST0000 complete [ 3884.515359] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3884.529287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3890.839331] Lustre: server umount lustre-MDT0000 complete [ 3892.704675] LustreError: 132578:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3892.739756] LustreError: 132578:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3895.794841] LustreError: 129947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782244911 with bad export cookie 6169102920092344137 [ 3895.813845] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3895.829816] LustreError: 129947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3896.195353] Lustre: server umount lustre-MDT0001 complete [ 3909.651799] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 3912.391377] Key type lgssc unregistered [ 3912.714734] LNet: 133756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3912.721739] LNetError: 133756:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3913.767310] LNet: Removed LNI 192.168.204.129@tcp [ 3914.678675] Key type .llcrypt unregistered [ 3914.687399] Key type ._llcrypt unregistered [ 3942.350181] Key type ._llcrypt registered [ 3942.353393] Key type .llcrypt registered [ 3942.445377] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 3943.867459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3943.886763] alg: No test for adler32 (adler32-zlib) [ 3945.027391] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 3945.283335] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 3946.975209] Key type lgssc registered [ 3948.283485] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3957.113412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3957.135067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3958.758967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3963.683396] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3968.993799] LustreError: 135171:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3969.010116] LustreError: 135171:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3974.124703] LustreError: 135172:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3974.624519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3975.362099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3980.785211] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3989.601608] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3995.962385] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4003.557227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4003.948084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4011.030240] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4019.175739] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4028.522185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4038.838178] Lustre: *** cfs_fail_loc=159, val=0*** [ 4054.533544] Lustre: lustre-MDT0000: Client 2ddcc215-890f-44cc-b056-2ef4a6b63743 (at 192.168.204.29@tcp) reconnecting [ 4057.096035] Lustre: *** cfs_fail_loc=159, val=0*** [ 4072.457212] Lustre: lustre-MDT0001: Client 2ddcc215-890f-44cc-b056-2ef4a6b63743 (at 192.168.204.29@tcp) reconnecting [ 4074.580610] Lustre: *** cfs_fail_loc=159, val=0*** [ 4074.587297] Lustre: Skipped 3 previous similar messages [ 4090.860494] Lustre: lustre-MDT0001: Client 2ddcc215-890f-44cc-b056-2ef4a6b63743 (at 192.168.204.29@tcp) reconnecting [ 4097.357393] Lustre: 138226:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4100.591679] Lustre: *** cfs_fail_loc=159, val=0*** [ 4100.593415] Lustre: Skipped 3 previous similar messages [ 4116.474039] Lustre: lustre-MDT0000: Client d6aee869-a3b6-4477-8e35-8f5968eb1168 (at 192.168.204.29@tcp) reconnecting [ 4118.593590] Lustre: *** cfs_fail_loc=159, val=0*** [ 4118.609208] Lustre: Skipped 13 previous similar messages [ 4134.432534] Lustre: lustre-MDT0000: Client d6aee869-a3b6-4477-8e35-8f5968eb1168 (at 192.168.204.29@tcp) reconnecting [ 4137.138747] Lustre: 138518:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4156.901325] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4156.911743] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4156.927035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4156.933637] Lustre: Skipped 1 previous similar message [ 4156.954762] Lustre: Skipped 1 previous similar message [ 4161.215717] Lustre: server umount lustre-OST0000 complete [ 4167.137174] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4167.140549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4167.162954] Lustre: Skipped 1 previous similar message [ 4171.423958] Lustre: server umount lustre-MDT0000 complete [ 4172.261474] LustreError: 135172:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4172.292532] LustreError: 135172:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4175.264335] LustreError: 135154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782245191 with bad export cookie 3446746628273923918 [ 4175.269836] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4175.277850] LustreError: 135154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4175.721798] Lustre: server umount lustre-MDT0001 complete [ 4187.678824] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 4191.562827] Key type lgssc unregistered [ 4191.951763] LNet: 139596:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4191.966430] LNetError: 139596:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4192.008443] LNet: Removed LNI 192.168.204.129@tcp [ 4193.123536] Key type .llcrypt unregistered [ 4193.127514] Key type ._llcrypt unregistered [ 4220.798821] Key type ._llcrypt registered [ 4220.807711] Key type .llcrypt registered [ 4220.956708] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 4222.315362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4222.484156] alg: No test for adler32 (adler32-zlib) [ 4223.668926] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 4223.995319] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 4225.727190] Key type lgssc registered [ 4227.425706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4237.214973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4237.252587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4238.838324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4244.046784] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4249.075801] LustreError: 141011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4249.107077] LustreError: 141011:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4254.180988] LustreError: 141012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4255.214498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4255.692807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4261.117179] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4270.677921] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4277.604889] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4285.552946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4286.010951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4293.015828] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4300.364633] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4302.254487] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4308.597103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4314.153838] Lustre: 143594:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4317.433021] Lustre: 143642:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4319.326342] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4320.352284] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4324.644884] Lustre: 143790:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4325.913067] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4327.067857] Lustre: 143901:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4334.565133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4334.572922] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4334.607892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4334.629938] Lustre: Skipped 1 previous similar message [ 4337.949868] Lustre: server umount lustre-OST0000 complete [ 4342.752422] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4342.762526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4342.770317] Lustre: Skipped 1 previous similar message [ 4342.774136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4344.802389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4344.803390] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4348.954653] Lustre: server umount lustre-MDT0000 complete [ 4349.923786] LustreError: 143770:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4349.948842] LustreError: 143770:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4353.757226] LustreError: 140993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782245369 with bad export cookie 11668285733887691140 [ 4353.761374] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4353.774307] LustreError: 140993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4354.277401] Lustre: server umount lustre-MDT0001 complete [ 4368.310268] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 4371.487781] Key type lgssc unregistered [ 4371.960262] LNet: 144957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4371.977522] LNetError: 144957:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4372.003654] LNet: Removed LNI 192.168.204.129@tcp [ 4373.268931] Key type .llcrypt unregistered [ 4373.272400] Key type ._llcrypt unregistered [ 4404.238349] Key type ._llcrypt registered [ 4404.241134] Key type .llcrypt registered [ 4404.378755] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 4405.497933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4405.532669] alg: No test for adler32 (adler32-zlib) [ 4406.567235] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 4406.814476] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 4408.543360] Key type lgssc registered [ 4409.552276] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4417.562816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4417.587758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4419.127105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4423.923953] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4428.260026] LustreError: 146371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4428.281216] LustreError: 146371:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4432.817396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4433.415283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4439.422638] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4449.008734] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4455.394536] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4462.739560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4463.117322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4464.184963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4470.230147] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4478.523453] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4478.676646] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4484.667422] Lustre: *** cfs_fail_loc=159, val=0*** [ 4501.012713] Lustre: lustre-MDT0000: Client ecf83e16-6db4-41fc-8c35-f08f4f4bdf50 (at 192.168.204.29@tcp) reconnecting [ 4509.153358] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4509.168925] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4509.179626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4509.219222] Lustre: Skipped 1 previous similar message [ 4514.278318] LustreError: 148291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4514.304239] LustreError: 148291:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4514.448420] Lustre: server umount lustre-OST0000 complete [ 4519.393934] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4519.396857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4519.405415] Lustre: Skipped 1 previous similar message [ 4519.422166] Lustre: Skipped 2 previous similar messages [ 4524.516355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4524.521708] Lustre: Skipped 1 previous similar message [ 4524.761609] Lustre: server umount lustre-MDT0000 complete [ 4529.218391] LustreError: 146352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782245545 with bad export cookie 4967209678780190159 [ 4529.219973] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4529.229740] LustreError: 146352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4529.536681] Lustre: server umount lustre-MDT0001 complete [ 4545.577717] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 4548.773050] Key type lgssc unregistered [ 4549.073073] LNet: 150040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4549.081306] LNetError: 150040:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4549.100092] LNet: Removed LNI 192.168.204.129@tcp [ 4550.046747] Key type .llcrypt unregistered [ 4550.052871] Key type ._llcrypt unregistered [ 4578.741163] Key type ._llcrypt registered [ 4578.743435] Key type .llcrypt registered [ 4578.881720] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 4580.107490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4580.120738] alg: No test for adler32 (adler32-zlib) [ 4581.184994] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 4581.425121] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 4583.143315] Key type lgssc registered [ 4584.526564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4593.853929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4593.891117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4595.344160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4601.245934] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4605.926110] LustreError: 151456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4605.955587] LustreError: 151456:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4611.062088] LustreError: 151457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4611.191104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4611.610825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4617.113977] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4625.386700] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4632.828653] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4641.874864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4642.617788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4651.635600] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4656.959775] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4660.395959] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4665.144242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4684.861691] Lustre: lustre-MDT0001: evicting cd1cfde9-76a2-4bc0-beca-6369405467ad (at 192.168.204.29@tcp) by administrative request [ 4684.873178] Lustre: Skipped 1 previous similar message [ 4716.513896] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4716.517209] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4716.530766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4716.546360] Lustre: Skipped 1 previous similar message [ 4716.563975] Lustre: Skipped 1 previous similar message [ 4719.801859] Lustre: server umount lustre-OST0000 complete [ 4724.193650] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4724.206741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4724.227886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4726.752933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4726.757199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4729.848136] Lustre: server umount lustre-MDT0000 complete [ 4731.884483] LustreError: 151452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4731.911963] LustreError: 151452:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4733.560077] LustreError: 151438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782245749 with bad export cookie 1986654374428289066 [ 4733.570265] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4733.855267] Lustre: server umount lustre-MDT0001 complete [ 4744.638706] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 4747.157495] Key type lgssc unregistered [ 4747.398731] LNet: 155529:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4747.407365] LNetError: 155529:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4747.424351] LNet: Removed LNI 192.168.204.129@tcp [ 4748.189705] Key type .llcrypt unregistered [ 4748.191791] Key type ._llcrypt unregistered [ 4826.040724] Key type ._llcrypt registered [ 4826.045414] Key type .llcrypt registered [ 4826.196429] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 4827.410738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4827.580946] alg: No test for adler32 (adler32-zlib) [ 4828.838514] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 4829.255722] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 4830.951192] Key type lgssc registered [ 4833.086984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4842.539964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4842.579979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4844.300218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4848.856764] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4854.763496] LustreError: 158019:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4854.794741] LustreError: 158019:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4859.713096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4859.876754] LustreError: 158018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4860.186157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4865.270647] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4874.184037] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4880.733081] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4888.723952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4889.306020] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4890.360630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4897.239848] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4904.691745] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4905.436870] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4916.193108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4916.207119] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4916.221511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4920.292684] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4920.295694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4920.296934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4921.568223] Lustre: server umount lustre-OST0000 complete [ 4926.945049] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4926.953155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4926.971812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4926.977954] Lustre: Skipped 1 previous similar message [ 4930.528781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4930.533133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4932.475964] Lustre: server umount lustre-MDT0000 complete [ 4935.648758] LustreError: 158018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4935.670311] LustreError: 158018:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4936.120990] LustreError: 157999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782245952 with bad export cookie 7363954093395209988 [ 4936.132807] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4936.137034] LustreError: 157999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4936.402561] Lustre: server umount lustre-MDT0001 complete [ 4948.096872] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 4951.433973] Key type lgssc unregistered [ 4951.789160] LNet: 161564:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4951.800445] LNetError: 161564:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4951.821082] LNet: Removed LNI 192.168.204.129@tcp [ 4952.694499] Key type .llcrypt unregistered [ 4952.696407] Key type ._llcrypt unregistered [ 4980.338271] Key type ._llcrypt registered [ 4980.343300] Key type .llcrypt registered [ 4980.432406] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 4981.598889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4981.626677] alg: No test for adler32 (adler32-zlib) [ 4982.750305] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 4983.022571] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 4984.719224] Key type lgssc registered [ 4985.888759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5002.237580] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 16:20:16 (1782246016) [ 5021.026771] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 5033.253336] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5071.739413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5071.771466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5072.981444] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5073.012692] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5073.106994] Lustre: lustre-MDT0000: new disk, initializing [ 5073.252583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5073.279154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5077.510503] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5087.452420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5087.592578] Lustre: 165846:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5087.691756] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5087.697958] Lustre: Skipped 1 previous similar message [ 5087.769516] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5087.801152] Lustre: lustre-MDT0001: new disk, initializing [ 5087.899041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5087.963336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5087.971290] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5092.476856] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5101.454981] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5107.214348] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5114.716359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5115.032590] Lustre: lustre-OST0000: new disk, initializing [ 5115.038204] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5115.047643] Lustre: 167816:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5115.167798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5121.589780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5121.620597] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5121.773881] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5121.829563] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5131.015938] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5142.505259] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5142.511809] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5142.522523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5142.545962] Lustre: Skipped 1 previous similar message [ 5142.575414] Lustre: Skipped 1 previous similar message [ 5147.620330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5147.629813] Lustre: Skipped 1 previous similar message [ 5147.920712] Lustre: server umount lustre-OST0000 complete [ 5152.738828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5152.741615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5152.766679] Lustre: Skipped 1 previous similar message [ 5152.782391] Lustre: Skipped 1 previous similar message [ 5157.863420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5157.874697] Lustre: Skipped 1 previous similar message [ 5166.047647] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5166.302725] Lustre: server umount lustre-MDT0000 complete [ 5170.499086] LustreError: 165841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782246186 with bad export cookie 2673235533106478455 [ 5170.501240] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5170.508455] LustreError: 165841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5170.784840] Lustre: server umount lustre-MDT0001 complete [ 5184.283989] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 5187.330669] Key type lgssc unregistered [ 5187.641827] LNet: 169721:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5187.653245] LNetError: 169721:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5187.674573] LNet: Removed LNI 192.168.204.129@tcp [ 5188.849977] Key type .llcrypt unregistered [ 5188.852110] Key type ._llcrypt unregistered [ 5213.892641] Key type ._llcrypt registered [ 5213.898258] Key type .llcrypt registered [ 5214.006707] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5214.867147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5214.953101] alg: No test for adler32 (adler32-zlib) [ 5215.953568] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 5216.108157] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 5217.807344] Key type lgssc registered [ 5218.843453] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5227.214298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5227.241300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5228.744831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5232.908764] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5238.311681] LustreError: 171139:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5238.338318] LustreError: 171139:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5242.949299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5243.308206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5247.963178] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5256.039125] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5263.552116] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5273.960264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5274.585189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5281.936125] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5289.249204] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5299.242184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5305.535714] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5305.548472] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5305.557725] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5305.562951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5305.575895] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5306.254495] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5306.268964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5306.281136] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5306.297428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.332507] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5307.076105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.848748] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5307.852908] Lustre: Skipped 1 previous similar message [ 5307.880087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5307.889179] Lustre: Skipped 1 previous similar message [ 5307.892464] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5307.896836] Lustre: Skipped 1 previous similar message [ 5307.899932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.911043] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5307.917242] Lustre: Skipped 1 previous similar message [ 5308.689647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5309.562432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.317757] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5310.325524] Lustre: Skipped 2 previous similar messages [ 5310.342320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5310.357733] Lustre: Skipped 2 previous similar messages [ 5310.361933] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5310.370351] Lustre: Skipped 2 previous similar messages [ 5310.375684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.385313] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5310.390705] Lustre: Skipped 2 previous similar messages [ 5311.205699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.069231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.940756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.744914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.539967] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5314.547171] Lustre: Skipped 4 previous similar messages [ 5314.557912] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5314.566494] Lustre: Skipped 4 previous similar messages [ 5314.570128] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5314.575323] Lustre: Skipped 4 previous similar messages [ 5314.582206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.599818] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5314.605335] Lustre: Skipped 4 previous similar messages [ 5315.296204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.133708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.926148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.765849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.535698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.364745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.163815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5321.112418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.036408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.048986] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5323.057221] Lustre: Skipped 9 previous similar messages [ 5323.084787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5323.100715] Lustre: Skipped 9 previous similar messages [ 5323.109961] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5323.126913] Lustre: Skipped 9 previous similar messages [ 5323.132878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.148929] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5323.152440] Lustre: Skipped 9 previous similar messages [ 5324.120900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5325.037991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.091141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.912771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.696107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.464816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.541723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.501642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.588321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.554932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.542148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.335283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.428501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.642765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.637758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.827236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.717470] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5339.726189] Lustre: Skipped 16 previous similar messages [ 5339.750282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5339.764636] Lustre: Skipped 16 previous similar messages [ 5339.767991] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5339.774431] Lustre: Skipped 16 previous similar messages [ 5339.778867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.793618] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5339.803779] Lustre: Skipped 16 previous similar messages [ 5340.800346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.765165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.742790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.589472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.625576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.649644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.694217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.945265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.850770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.947347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.035265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.084282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.086902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.962216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.880371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.793420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.654048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.546442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.296188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.165490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.094644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.006270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.021473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.948766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.014778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.839571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.833533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.726384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.744101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.790541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.790475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.769971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.717755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.606548] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5372.612501] Lustre: Skipped 33 previous similar messages [ 5372.627548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5372.639887] Lustre: Skipped 33 previous similar messages [ 5372.643446] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5372.648597] Lustre: Skipped 33 previous similar messages [ 5372.651613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.661099] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5372.671120] Lustre: Skipped 33 previous similar messages [ 5373.559032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.443380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.524952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.565850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.530037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.446461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.463656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.456459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.624399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.764421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.789253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.795836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.931545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.120400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.308377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.723722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.748535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.732477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.623740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.542654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.327914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.306317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.226930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.124109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.100026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.129714] LustreError: 171160:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5399.138162] LustreError: 171160:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5399.144246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.945178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.941655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.812920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.717064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.584421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.475346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.345458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.411442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.232850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.102240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.907527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.840400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.847854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.778406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.625526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.509471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.409701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.377720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.296429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.106176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.985569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.884922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.827947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.700203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.595794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.696740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.558983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.429389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.425339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.473967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.556857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.511560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.441500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.333439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.360551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.461779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.714591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.896223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.932724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.919330] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5436.928967] Lustre: Skipped 65 previous similar messages [ 5436.948632] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5436.957670] Lustre: Skipped 65 previous similar messages [ 5436.961870] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5436.965347] Lustre: Skipped 65 previous similar messages [ 5436.975213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.984663] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5436.990264] Lustre: Skipped 65 previous similar messages [ 5437.846399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.892991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.754486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.768784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.697596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.901231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.096718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.226656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.438506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.464326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.834075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.779329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.943464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.948595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.891241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.897841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.927809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.918806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.011465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.092626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.073988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.189509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.255111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.315986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.490108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.487673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.507949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.560505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.871050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.994414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.943200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.112262] LustreError: 171160:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5471.119193] LustreError: 171160:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5471.132080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.132454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.127738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.930822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.968710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.826187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.607323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.630042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.518745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.728680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.733264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.928259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.847045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.914851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.793846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.741518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.615384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.582700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.551314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.363831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.098676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.002389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.734217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.696692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.430579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.285055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.100806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.911108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.656391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.483360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.435483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.452832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.516926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.461806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.421772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.352596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.256855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.161780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.152923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.071536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.215651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.150053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.057670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.971941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.830714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.717721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.659205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.540241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.505497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.451038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.522643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.478398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.391332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.265316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.106439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.007045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.041970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.035630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.997704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.089373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.266534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.506565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.589992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.552867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.463318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.583055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.502889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.514625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.567733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.576489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.771372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.764236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.833739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.850521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.819968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.846229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.819345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.991596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.127588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.957996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.164744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.140970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.332681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.379390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.427784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.466537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.803439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.985235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.516213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.769303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.970744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.247700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.544953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.444752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.919491] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5565.931690] Lustre: Skipped 125 previous similar messages [ 5565.984801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5566.017137] Lustre: Skipped 125 previous similar messages [ 5566.044066] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5566.063963] Lustre: Skipped 125 previous similar messages [ 5566.078555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.103701] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5566.120923] Lustre: Skipped 125 previous similar messages [ 5567.549944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.714592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.625813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.782536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.878366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.968195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.827411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.975625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.993283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.173963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.274537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.279190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.246666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.193490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.073895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.874150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.053874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.035126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.082301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.083376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.890328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.884226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.846076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.938247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.821534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.829835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.857567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.991865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.964236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.060710] LustreError: 171160:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5598.068119] LustreError: 171160:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5598.084522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.101688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.098791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.002877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.051166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.061786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.941806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.885793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.965444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.035860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.129615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.079262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.104325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.039964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.951467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.892059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.813749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.821517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.753778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.766232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.718152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.710897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.662317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.635671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.607094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.671517] LustreError: 171160:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5622.688734] LustreError: 171160:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5622.692730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.704775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.717419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.707736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.732651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.913600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.811137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.803520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.865685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.858270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.815163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.621727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.575504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.376742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.499112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.619127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.564896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.661290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.585442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.450673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.287359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.150767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.154272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.170396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.033438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.013509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.023044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.965686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.857346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.710380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.506499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.485852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.581274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.417090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.403640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.403929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.565084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.736614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.796069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.758747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.818925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.037857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.260370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.224948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.484799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.629983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.755651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.875192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.008377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.906609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.428668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.607709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.714270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.761611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.218353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.655066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.931124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.195123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.396696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.404514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.489684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.431831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.526456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.489724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.729983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.114736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.469467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.623379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.792775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.634701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.611820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.866241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.285286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.951569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.382650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.466113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.688196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.442888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.258257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.684553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.990667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.956826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.048478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.946606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.164754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.247187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.164796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.098098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.095534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.069682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.953090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.035685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.921362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.839669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.769544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.699774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.483686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.355754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.476280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.389439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.339813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.366748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.310222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.495705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.515949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.223402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.068105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.272615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.669449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.653967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.718697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.074732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.426694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.505785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.660474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.308468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.708654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.668243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.586207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.823901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.424255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.769333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.858810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.639648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.164499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.344744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.633259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.809876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.873361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.082757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.063987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.274860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.387797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.526963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.486537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.326106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.379801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.304696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.245911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.225183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.021943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.933990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.853559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.835891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.768356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.617509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.453563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.296433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.160183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.967325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.761726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.695635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.656759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.550666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.479493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.426782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.214928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.238724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.260586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.158953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.082386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.998281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.826298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.661586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.451088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.452982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.296768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.070757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.944126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.889529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.629615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.625504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.530455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.670970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.716677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.798722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.665590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.612590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.554706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.536813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.722860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.599100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.542837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.522499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.598602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.554255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.764564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.931913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.935844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.175705] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5823.185484] Lustre: Skipped 243 previous similar messages [ 5823.208833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5823.220696] Lustre: Skipped 243 previous similar messages [ 5823.229410] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5823.238644] Lustre: Skipped 243 previous similar messages [ 5823.246295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.261929] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5823.278275] Lustre: Skipped 243 previous similar messages [ 5824.547869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.901620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.272610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.164811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.084494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.016887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.010625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.226497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.320317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.302063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.334945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.425719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.637458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.553891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.839697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.955201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.170526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.469296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.481384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.720904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.666938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.198421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.357473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.617454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.919809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.245626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.552601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.655264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.453965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.742976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.732645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.854890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.921741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.889653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.347504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.396787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.454676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.069036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.533556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.936578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.202258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.400186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.500974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.133271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.690115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.910321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.347758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.985601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.320845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.573665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.839528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.063189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.526876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.777267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.218660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.866164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.130835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.076678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.131152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.334866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.465129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.650959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.948048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.949461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.280964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.314887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.375633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.360668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.437606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.397541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.600686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.778559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.009990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.994628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.020420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.031181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.083965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.041324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.075481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.013978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.262106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.283494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.304423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.462374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.728511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.892947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.982151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.180066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.041256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.076938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.152555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.316248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.793296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.948514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.927607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.068686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.096113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.230190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.585274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.535786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.744201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.720324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.795356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.671242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.697091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.553266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.491618] LustreError: 171160:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5947.497506] LustreError: 171160:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5947.498741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.502409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.605933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.546508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.460475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.584764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.467599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.522885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.621372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.573217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.500664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.447693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.559095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.631250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.850745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.910635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.795128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.804807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.773312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.778640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.763706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.830272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.614049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.741304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.804217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.884851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.697201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.610638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.497436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.477313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.626741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.613048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.685933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.787197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.905521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.844217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.848267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.885878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.005652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.008610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.263279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.398405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.365563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.331769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.198671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.116743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.035694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.037955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.048622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.994292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.071645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.957151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.017839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.937258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.926934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.919869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.902974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.849900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.820475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.932217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.102286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.115981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.108471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.161968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.999509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.989805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.009816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.970222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.954758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.915847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.780991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.885976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.854707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.055822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.998930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.098231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.213450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.094595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.144534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.117428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.247597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.424227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.444361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.349349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.231536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.310480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.332688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.287680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.289400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.412772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.350275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.401642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.346324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.254738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.179552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.136115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.945789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.056652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.951722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.778817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.725533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.685494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.745718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.680422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.671805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.599648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.545868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.395140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.262630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.340873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.521497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.478728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.503975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.468934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.674700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.816514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.554155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.555756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.775415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.037735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.118103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.513218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.774766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.100752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.101130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.107866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.006974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.050686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.043274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.104503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.084469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.008185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.089329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.049533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.101596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.247983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.165179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.104405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.220520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.791475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.204948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.268664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.410936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.543024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.930129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.351688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.349043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.894234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.150673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.819384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.959529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.926599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.139806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.292385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.452473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.441344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.500357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.439748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.581968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.855545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.979690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.913105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.958564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.050766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.134218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.126969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.107295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.897274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.772486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.712268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.696613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.748842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.626878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.534698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.478312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.509634] LustreError: 171160:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6128.518924] LustreError: 171160:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6128.526390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6129.655936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.589730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.443085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.329923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.348571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.270704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.215715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.101282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.048167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.134799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.143017] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6140.253659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.185631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.158586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.086697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.019402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.788639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.619916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.449887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.488744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.468113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.405252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6150.293103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.229352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.167461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6153.082894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6153.977340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6154.952901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.795630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.740978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6157.654075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6158.514200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6159.268856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6160.244374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.058695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.964334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.744586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6163.759265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6164.670175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.729491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.773539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6167.918921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6169.133912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6170.152364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6171.045947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6171.956430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6172.963775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6173.847258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.862446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6175.725907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6176.723735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.601267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6178.624462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6179.434056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6180.312289] LustreError: 171160:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6180.324609] LustreError: 171160:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6180.345309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6181.305027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6182.372468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.242963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.035706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.918169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6185.685408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6186.379059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.164139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.044422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.869499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6189.719914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.650797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.481450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6192.232161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6193.150286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.048768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.957548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.999257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.919172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.907815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.839482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6199.784127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6200.677625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6201.560399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6202.464295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.413751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6204.396287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6205.493157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.562653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6207.605814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6208.728608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6209.685155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6210.659651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6211.572339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6212.647879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6213.562981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6214.476878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6215.470730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6216.674045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6217.816846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6218.876449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6219.779956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6220.869216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6221.941616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6223.210628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6224.374891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6225.288685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6226.112287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6227.041475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6228.120185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6228.987584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6229.926337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6230.928919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6232.104509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6233.362815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6234.410790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6235.561912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6236.447703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6237.563785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6238.536933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6239.721528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6240.892338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6241.738090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6242.729969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6243.630499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6244.583570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6245.749688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6246.507250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6247.518660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6248.573566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6249.578593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6250.530887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6251.568965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6252.760792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6253.918716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6255.062476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6256.136230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6257.063285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6257.876031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6258.709783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6259.525674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6260.418931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6261.273761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6262.186428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6263.140578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6264.051788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6264.842755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6265.746700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6266.525272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6267.326294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6268.183525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6269.121274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6270.287237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6271.452675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6272.742368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6273.914140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6275.079633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6276.230921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6277.390377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6278.657810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6279.605431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6280.573415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6281.611641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6282.599637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6283.507187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6284.645670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6285.604990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6286.749682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6287.708855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6288.659050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6289.616301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6290.697286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6291.676805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6292.886934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6293.893252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6294.845290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6295.757507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6296.843481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6297.925933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6299.090887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6300.278098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6301.362343] LustreError: 171160:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6301.380279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6301.380483] LustreError: 171160:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6302.503132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6303.579460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6304.656144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6305.768778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6306.952357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6307.999144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6308.792868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6309.692067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6310.653103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6311.754577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6312.822764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6313.908897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6315.064842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6315.997048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6317.061352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6318.358601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6319.481083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6320.652051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6321.775549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6322.921404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6323.911448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6324.864711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6325.766554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6326.893806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6327.901135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6328.862691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6329.771113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6330.679773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6331.636460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6332.674955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6333.595065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6334.579403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6827.233797] LustreError: 172964:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6827.250452] LustreError: 171160:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 7628.796505] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7628.814852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7628.849329] Lustre: Skipped 491 previous similar messages [ 7628.855908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7630.821381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7630.827931] Lustre: Skipped 1 previous similar message [ 7632.691833] Lustre: server umount lustre-OST0000 complete [ 7639.531967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7639.546426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7646.179688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7646.187245] Lustre: Skipped 3 previous similar messages [ 7651.297475] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7651.298722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7651.323048] Lustre: Skipped 1 previous similar message [ 7651.729610] Lustre: server umount lustre-MDT0000 complete [ 7655.410608] LustreError: 171140:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7655.442827] LustreError: 171140:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7656.105823] LustreError: 171120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782248672 with bad export cookie 14460538755412666437 [ 7656.122890] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7656.133265] LustreError: 171120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7656.572892] Lustre: server umount lustre-MDT0001 complete [ 7669.939556] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 7672.758621] Key type lgssc unregistered [ 7673.054948] LNet: 226475:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7673.077729] LNetError: 226475:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7673.098600] LNet: Removed LNI 192.168.204.129@tcp [ 7674.130123] Key type .llcrypt unregistered [ 7674.135333] Key type ._llcrypt unregistered [ 7702.177721] Key type ._llcrypt registered [ 7702.181480] Key type .llcrypt registered [ 7702.319828] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 7703.689740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7703.714993] alg: No test for adler32 (adler32-zlib) [ 7705.100699] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 7705.497429] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 7707.256247] Key type lgssc registered [ 7708.776337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7717.914831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7717.932451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7719.496736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7725.464253] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7729.639091] LustreError: 227890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7729.669145] LustreError: 227890:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7734.752629] LustreError: 227891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7735.420517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7735.901740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7742.061796] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7751.613560] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7758.087818] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7765.941496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7766.544603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7773.175761] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7781.240099] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7789.201471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7797.986310] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7800.095815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7800.107093] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7800.123838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7800.138143] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7800.138267] LustreError: 227910:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7803.377648] Lustre: lustre-OST0000: Not available for connect from 192.168.204.29@tcp (stopping) [ 7804.898278] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7804.924865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7804.937637] Lustre: Skipped 1 previous similar message [ 7808.442073] Lustre: lustre-OST0000: Not available for connect from 192.168.204.29@tcp (stopping) [ 7808.875333] Lustre: server umount lustre-OST0000 complete [ 7820.231874] LustreError: 227910:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7845.468093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7845.814084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7851.910578] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7859.093744] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7861.169236] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7868.393415] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c2186d43000.ost_server_uuid 50 [ 7869.304339] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7869.323111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7869.323960] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7869.343746] Lustre: Skipped 1 previous similar message [ 7870.341340] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8c2186d43000.ost_server_uuid in FULL state after 0 sec [ 7879.668274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7879.669481] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7879.678984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7879.700514] Lustre: Skipped 2 previous similar messages [ 7879.714888] Lustre: Skipped 1 previous similar message [ 7881.506173] Lustre: server umount lustre-OST0000 complete [ 7889.382955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7889.394676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7889.417644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7899.108595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7899.124519] Lustre: Skipped 4 previous similar messages [ 7900.127839] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7900.369605] Lustre: server umount lustre-MDT0000 complete [ 7903.897740] LustreError: 228624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782248919 with bad export cookie 15582548145535139876 [ 7903.901962] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7903.913180] LustreError: 228624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7904.206164] Lustre: server umount lustre-MDT0001 complete [ 7914.567267] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 7917.444702] Key type lgssc unregistered [ 7917.769965] LNet: 233117:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7917.786621] LNetError: 233117:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7917.802796] LNet: Removed LNI 192.168.204.129@tcp [ 7918.759312] Key type .llcrypt unregistered [ 7918.761367] Key type ._llcrypt unregistered [ 7953.627574] Key type ._llcrypt registered [ 7953.635365] Key type .llcrypt registered [ 7953.839329] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 7955.429847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7955.519092] alg: No test for adler32 (adler32-zlib) [ 7956.664043] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 7956.952572] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 7958.655486] Key type lgssc registered [ 7960.108508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7969.769808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7969.813977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7971.386821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7976.752141] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7981.537816] LustreError: 234847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7981.557714] LustreError: 234847:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7986.407122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7986.657132] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7986.896413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7992.080641] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 7995.501144] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8001.750711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8002.113178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8007.769899] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8016.431226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 8016.440306] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8016.608157] Lustre: lustre-OST0001: new disk, initializing [ 8016.614500] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8016.620895] Lustre: 237155:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8016.724474] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8022.752783] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8026.231527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8026.243505] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8026.398981] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8034.275451] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8041.383275] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8057.327872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8057.338588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8060.424039] Lustre: server umount lustre-MDT0000 complete [ 8062.437699] LustreError: 234842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8062.466819] LustreError: 234842:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8064.515858] LustreError: 234827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782249080 with bad export cookie 11673564987021111071 [ 8064.526035] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8064.536155] LustreError: 234827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8064.904541] Lustre: server umount lustre-MDT0001 complete [ 8078.433784] Lustre: server umount lustre-OST0000 complete [ 8092.303204] Lustre: server umount lustre-OST0001 complete [ 8120.783851] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 8132.027798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8132.459190] LustreError: 240263:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8132.563914] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8136.588941] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8137.707049] LustreError: 240264:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8142.817762] LustreError: 240263:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8143.855931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8144.226178] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8148.626348] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8151.641967] Lustre: 241405:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8158.876471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8166.580260] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8169.575040] LustreError: 241760:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8169.614828] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 8176.697656] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8177.060590] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8177.066620] Lustre: Skipped 1 previous similar message [ 8178.107473] Lustre: mylustre-OST0001: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33) [ 8178.109815] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49760) [ 8184.310224] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8192.292807] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8196.562226] Lustre: 243277:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8208.871388] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8208.881310] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8208.883226] Lustre: Skipped 6 previous similar messages [ 8208.903292] Lustre: Skipped 6 previous similar messages [ 8213.646910] Lustre: server umount mylustre-MDT0000 complete [ 8213.984688] LustreError: 240996:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8214.001207] LustreError: 240996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8217.561361] LustreError: 240240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782249233 with bad export cookie 11673564987021113920 [ 8217.568727] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8217.571763] LustreError: 240240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8217.988721] Lustre: server umount mylustre-MDT0001 complete [ 8232.137310] Lustre: server umount mylustre-OST0000 complete [ 8235.487683] Lustre: 234110:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782249235/real 1782249235] req@ffff8cd3c66f4380 x1868823498221056/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782249251 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8235.551822] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8237.058274] Lustre: server umount mylustre-OST0001 complete [ 8263.003615] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 8273.924395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8274.467277] LustreError: 245588:0:(ldlm_lib.c:1179:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8274.561873] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8280.330541] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8291.630602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8297.188681] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8300.486078] Lustre: 246733:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8308.201567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8315.896334] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8316.797632] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:161) [ 8319.941631] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49792) [ 8325.488538] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8331.270614] Lustre: tfs-OST0001: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:65) [ 8333.057584] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8342.147613] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8346.187542] Lustre: 248605:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8356.835739] Lustre: tfs-MDT0000-lwp-OST0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8356.840831] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 8356.859159] Lustre: Skipped 2 previous similar messages [ 8356.877480] Lustre: Skipped 3 previous similar messages [ 8360.467551] Lustre: server umount tfs-MDT0000 complete [ 8361.966444] LustreError: 246323:0:(ldlm_lib.c:1179:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8362.000546] LustreError: 246323:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 8364.464519] LustreError: 247110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782249380 with bad export cookie 11673564987021116097 [ 8364.476857] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8364.485471] LustreError: 247110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8364.923706] Lustre: server umount tfs-MDT0001 complete [ 8379.916439] Lustre: server umount tfs-OST0000 complete [ 8382.752090] Lustre: 234111:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782249382/real 1782249382] req@ffff8cd2c25f8e00 x1868823498274944/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782249398 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8382.779164] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8382.792666] Lustre: Skipped 1 previous similar message [ 8383.866526] Lustre: server umount tfs-OST0001 complete [ 8407.666513] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 8422.128129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8422.918207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8422.927907] Lustre: Skipped 3 previous similar messages [ 8428.177563] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8438.599129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8444.129046] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8447.415422] Lustre: 251998:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8454.335467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8461.448598] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8465.081534] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:193) [ 8468.159427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49824) [ 8471.529971] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8477.196357] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:97) [ 8479.132143] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8487.548466] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8491.306390] Lustre: 253867:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8500.144614] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 17:18:33 (1782249513) [ 8518.126712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8518.135289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8518.141052] Lustre: Skipped 2 previous similar messages [ 8518.146961] Lustre: Skipped 1 previous similar message [ 8522.743634] Lustre: server umount lustre-MDT0000 complete [ 8523.244448] LustreError: 250850:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8523.256859] LustreError: 250850:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 8526.654071] LustreError: 251601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782249542 with bad export cookie 11673564987021118015 [ 8526.654553] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8526.676617] LustreError: 251601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8528.367856] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8528.377896] Lustre: Skipped 1 previous similar message [ 8533.230429] Lustre: server umount lustre-MDT0001 complete [ 8543.588147] Lustre: server umount lustre-OST0000 complete [ 8554.208150] Lustre: server umount lustre-OST0001 complete [ 8561.321929] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 8573.345000] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 8619.871493] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 8619.871493] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8619.871493] [ 8619.936966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8620.209782] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8620.255146] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8620.377597] Lustre: lustre-MDT0000: new disk, initializing [ 8620.530241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8620.541296] Lustre: Skipped 3 previous similar messages [ 8620.570791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8626.568442] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8640.783317] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 8640.783317] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8640.783317] [ 8640.864508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8641.058781] Lustre: 257057:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8641.127526] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8641.131749] Lustre: Skipped 1 previous similar message [ 8641.283860] Lustre: lustre-MDT0001: new disk, initializing [ 8641.567454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8641.594575] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8648.602103] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8660.196766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8660.629457] Lustre: lustre-OST0000: new disk, initializing [ 8660.633687] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8660.639590] Lustre: 258639:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8662.162888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8662.172101] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8662.300381] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8668.660989] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8680.573287] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8680.785351] Lustre: lustre-OST0001: new disk, initializing [ 8680.788795] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8680.795493] Lustre: 259512:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8682.657463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8682.670914] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8682.778416] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8688.597482] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8700.733375] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 17:21:54 (1782249714) [ 8717.569793] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 17:22:11 (1782249731) [ 8734.178129] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8734.191593] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8734.199424] LustreError: Skipped 1 previous similar message [ 8734.206529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8734.226123] Lustre: Skipped 4 previous similar messages [ 8734.242939] Lustre: Skipped 3 previous similar messages [ 8739.839035] Lustre: server umount lustre-OST0000 complete [ 8750.849647] Lustre: server umount lustre-MDT0000 complete [ 8755.109704] LustreError: 257047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782249771 with bad export cookie 11673564987021119786 [ 8755.111916] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8755.127327] LustreError: 257047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8761.586775] Lustre: server umount lustre-MDT0001 complete [ 8773.628126] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 8786.502374] Lustre: server umount lustre-OST0001 complete [ 8794.831772] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 8812.302831] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 8853.910709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8854.171697] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8854.198916] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8854.304264] Lustre: lustre-MDT0000: new disk, initializing [ 8854.432814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8859.948844] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8871.812951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8871.930096] Lustre: 264067:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8871.989380] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8871.993346] Lustre: Skipped 1 previous similar message [ 8872.054464] Lustre: lustre-MDT0001: new disk, initializing [ 8872.197546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8872.213288] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8877.965661] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8889.863228] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8896.653533] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8903.850329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8904.178464] Lustre: lustre-OST0000: new disk, initializing [ 8904.184343] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8904.195241] Lustre: 266036:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8904.320912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8904.330417] Lustre: Skipped 5 previous similar messages [ 8905.378542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8905.404909] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8905.537182] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8911.264656] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 8921.212080] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8931.305433] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8931.310836] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8931.323259] LustreError: Skipped 1 previous similar message [ 8931.330790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8931.344427] Lustre: Skipped 5 previous similar messages [ 8931.359952] Lustre: Skipped 10 previous similar messages [ 8934.583261] Lustre: server umount lustre-OST0000 complete [ 8954.335535] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8955.006798] Lustre: server umount lustre-MDT0000 complete [ 8956.903275] LustreError: 264076:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8956.975878] LustreError: 264076:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 8960.336737] LustreError: 264863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782249976 with bad export cookie 11673564987021121263 [ 8960.338700] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8960.352139] LustreError: 264863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8960.818282] Lustre: server umount lustre-MDT0001 complete [ 8974.485433] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 8978.224939] Key type lgssc unregistered [ 8978.525824] LNet: 267800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8978.533341] LNetError: 267800:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8978.569898] LNet: Removed LNI 192.168.204.129@tcp [ 8979.656984] Key type .llcrypt unregistered [ 8979.658955] Key type ._llcrypt unregistered [ 9007.679930] Key type ._llcrypt registered [ 9007.682575] Key type .llcrypt registered [ 9007.798915] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 9008.975953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9009.040978] alg: No test for adler32 (adler32-zlib) [ 9010.089939] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 9010.337574] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 9012.017463] Key type lgssc registered [ 9013.159305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9021.462716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9021.495953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9023.006119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9028.452218] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9032.163118] LustreError: 269263:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9032.178346] LustreError: 269263:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9037.286621] LustreError: 269264:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9038.648470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9039.193962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9044.224078] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9052.702840] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9058.816534] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9066.199397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9066.636110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9067.696218] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 9072.819127] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9079.886250] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9080.798194] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 9083.361928] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9083.375342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9083.395673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9084.456193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9084.466683] Lustre: Skipped 1 previous similar message [ 9089.507157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9089.515577] Lustre: Skipped 1 previous similar message [ 9094.633715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9094.642266] Lustre: Skipped 1 previous similar message [ 9097.183874] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9097.303675] Lustre: server umount lustre-OST0000 complete [ 9104.864907] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9104.905264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9107.316609] Lustre: server umount lustre-MDT0000 complete [ 9109.984357] LustreError: 269263:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9110.016590] LustreError: 269263:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9111.193379] LustreError: 269245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782250127 with bad export cookie 16922641911622843632 [ 9111.198562] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9111.205583] LustreError: 269245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9111.512601] Lustre: server umount lustre-MDT0001 complete [ 9121.559287] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 9123.948612] Key type lgssc unregistered [ 9124.257838] LNet: 272850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9124.276489] LNetError: 272850:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9125.352070] LNet: Removed LNI 192.168.204.129@tcp [ 9126.254899] Key type .llcrypt unregistered [ 9126.260523] Key type ._llcrypt unregistered [ 9142.412436] Key type ._llcrypt registered [ 9142.415213] Key type .llcrypt registered [ 9142.525181] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 9143.682201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9143.731133] alg: No test for adler32 (adler32-zlib) [ 9144.924898] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 9145.251848] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 9146.983218] Key type lgssc registered [ 9148.158890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9159.488755] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9159.506554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9175.263495] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9180.447392] LustreError: 274045:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.129@tcp: failed processing log, type 4: rc = -110 [ 9207.071237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9213.652513] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9221.908712] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9224.885442] Lustre: server umount lustre-OST0000 complete [ 9244.666758] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 9247.967881] Key type lgssc unregistered [ 9248.500626] LNet: 275677:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9248.512297] LNetError: 275677:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9248.539360] LNet: Removed LNI 192.168.204.129@tcp [ 9249.731800] Key type .llcrypt unregistered [ 9249.737282] Key type ._llcrypt unregistered [ 9298.585802] Key type ._llcrypt registered [ 9298.593088] Key type .llcrypt registered [ 9298.737740] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 9300.508528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9300.554518] alg: No test for adler32 (adler32-zlib) [ 9301.881520] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 9302.266832] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 9303.997543] Key type lgssc registered [ 9306.290715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9323.922731] loop0: detected capacity change from 0 to 204800000 [ 9324.164696] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9324.172272] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9324.436118] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9325.527664] loop0: detected capacity change from 0 to 204800000 [ 9325.638277] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9325.650973] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9325.908293] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9326.927771] loop0: detected capacity change from 0 to 204800000 [ 9327.116809] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9327.134741] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9327.273573] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9328.456478] loop0: detected capacity change from 0 to 204800000 [ 9328.623566] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9329.957203] loop0: detected capacity change from 0 to 204800000 [ 9329.978639] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9334.132776] loop0: detected capacity change from 0 to 204800000 [ 9334.211480] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9338.709192] loop0: detected capacity change from 0 to 204800000 [ 9338.748221] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9343.825671] loop0: detected capacity change from 0 to 204800000 [ 9343.887458] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9348.049779] loop0: detected capacity change from 0 to 204800000 [ 9348.097779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9348.140685] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9348.241396] Lustre: 279098:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9348.251209] Lustre: 279098:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 9348.283692] Lustre: 279098:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 9348.301616] Lustre: 279098:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 9348.337816] Lustre: 279098:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9348.357839] Lustre: 279098:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9348.387446] Lustre: 279098:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9348.412576] Lustre: 279098:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 9348.418363] Lustre: 279098:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 9348.424807] Lustre: 279098:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 9348.430419] Lustre: 279098:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 9348.435893] Lustre: 279098:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 9348.441755] Lustre: 279098:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 9348.449972] Lustre: 279098:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 9355.422283] Lustre: server umount MGS complete [ 9356.497434] loop0: detected capacity change from 0 to 204800000 [ 9356.606253] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9357.055915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9357.113730] Lustre: lustre-MDT0000: Aborting client recovery [ 9357.906896] loop1: detected capacity change from 0 to 204800000 [ 9358.088075] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9358.240937] Lustre: 279528:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9358.250444] Lustre: 279528:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 9358.259314] Lustre: 279528:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9358.268739] Lustre: 279528:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9358.290448] Lustre: 279528:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 9358.303061] Lustre: 279528:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9358.312645] Lustre: 279528:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 9358.330299] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9358.482071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9358.520796] Lustre: lustre-MDT0001: Aborting client recovery [ 9359.475764] loop2: detected capacity change from 0 to 204800000 [ 9359.659666] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9359.779556] Lustre: 279608:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9359.838736] Lustre: 279608:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9359.854392] Lustre: 279608:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9359.869391] Lustre: 279608:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9359.882894] Lustre: 279608:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9359.902542] Lustre: 279608:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9359.914935] Lustre: 279608:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9360.114364] Lustre: Found index 0 for lustre-OST0000, updating log [ 9360.148428] Lustre: 279627:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9360.236140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9360.251950] Lustre: lustre-OST0000: Aborting recovery [ 9361.183154] loop3: detected capacity change from 0 to 204800000 [ 9361.288732] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9361.443789] Lustre: 279693:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9361.458193] Lustre: 279693:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9361.475348] Lustre: 279693:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9361.483863] Lustre: 279693:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9361.496459] Lustre: 279693:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9361.504939] Lustre: 279693:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9361.513669] Lustre: 279693:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9361.549082] Lustre: Found index 1 for lustre-OST0001, updating log [ 9361.594253] Lustre: 279699:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9361.704134] Lustre: lustre-OST0001: Aborting recovery [ 9366.847272] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9366.929550] LustreError: 279472:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 9366.958475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9367.010669] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 9367.047949] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 9368.897663] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 9368.919393] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 9371.578427] LustreError: 279439:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 11904us: rc = -2 [ 9372.130269] LustreError: 279440:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 558us: rc = -2 [ 9372.161525] LustreError: 279440:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 21 previous similar messages [ 9389.539922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9389.541080] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9389.542247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9389.578357] Lustre: Skipped 3 previous similar messages [ 9394.656426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9394.669483] Lustre: Skipped 6 previous similar messages [ 9399.263144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9399.493905] Lustre: server umount lustre-MDT0000 complete [ 9399.778582] LustreError: 279438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9399.810315] LustreError: 279438:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9400.550675] LustreError: 279423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782250416 with bad export cookie 9251725561609211259 [ 9400.572116] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9400.575725] LustreError: 279423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9404.897074] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9404.899368] LustreError: 279436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9404.907652] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9404.913774] Lustre: Skipped 1 previous similar message [ 9404.933066] LustreError: 279436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9405.939087] LustreError: 279633:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9405.959941] LustreError: 279633:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9406.958582] Lustre: server umount lustre-MDT0001 complete [ 9422.303390] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9422.484600] Lustre: server umount lustre-OST0000 complete [ 9429.751681] Lustre: server umount lustre-OST0001 complete [ 9438.085576] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 17:34:12 (1782250452) [ 9465.646882] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 9474.910587] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 9511.608914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9511.892881] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9511.921654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9512.051950] Lustre: lustre-MDT0000: new disk, initializing [ 9512.186567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9512.197547] Lustre: Skipped 1 previous similar message [ 9512.225497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9517.089252] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9527.586326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9527.713195] Lustre: 283765:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9527.779421] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9527.783138] Lustre: Skipped 1 previous similar message [ 9527.901445] Lustre: lustre-MDT0001: new disk, initializing [ 9528.050716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9528.099317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9528.115133] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9533.462950] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9543.279549] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9549.266407] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9556.514549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9556.819894] Lustre: lustre-OST0000: new disk, initializing [ 9556.823411] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9556.832623] Lustre: 285734:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9556.915032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9563.250652] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9565.276492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9565.289835] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9565.455529] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9572.591082] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9577.835515] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9591.289467] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9595.338946] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9601.221738] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9642.474069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9642.477482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9642.477976] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9642.494383] Lustre: Skipped 1 previous similar message [ 9642.507498] Lustre: Skipped 4 previous similar messages [ 9646.701105] Lustre: server umount lustre-OST0000 complete [ 9651.171360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9651.181345] LustreError: Skipped 1 previous similar message [ 9651.187911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9651.202674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9656.481208] Lustre: server umount lustre-MDT0000 complete [ 9657.825232] LustreError: 284956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9657.850897] LustreError: 284956:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9660.210665] LustreError: 284559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782250676 with bad export cookie 9251725561609213954 [ 9660.214269] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9660.220941] LustreError: 284559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9660.599322] Lustre: server umount lustre-MDT0001 complete [ 9670.551213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9675.059572] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9680.496154] Lustre: server umount MGS complete [ 9692.941748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9693.446905] LustreError: 289600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9693.474385] LustreError: 289613:0:(qmt_pool.c:1645:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9693.535700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9698.358583] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9703.919468] LustreError: 289601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9703.935315] LustreError: 289601:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9708.124052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9713.506864] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9722.235516] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9728.460738] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9736.166801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9736.535319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9736.541929] Lustre: Skipped 1 previous similar message [ 9743.358344] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9752.027785] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9779.683434] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9779.693452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9779.706724] Lustre: Skipped 1 previous similar message [ 9779.719832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9779.732844] Lustre: Skipped 2 previous similar messages [ 9785.219955] Lustre: server umount lustre-OST0000 complete [ 9790.432845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9790.450319] LustreError: Skipped 1 previous similar message [ 9790.453729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9790.474503] Lustre: Skipped 1 previous similar message [ 9795.758477] Lustre: server umount lustre-MDT0000 complete [ 9796.064255] LustreError: 290329:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9796.084254] LustreError: 290329:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9799.673218] LustreError: 292438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782250815 with bad export cookie 9251725561609215802 [ 9799.679244] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9799.688201] LustreError: 292438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9800.017534] Lustre: server umount lustre-MDT0001 complete [ 9812.755861] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 9815.596398] Key type lgssc unregistered [ 9815.939291] LNet: 293545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9815.962431] LNetError: 293545:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9815.990251] LNet: Removed LNI 192.168.204.129@tcp [ 9817.047982] Key type .llcrypt unregistered [ 9817.049855] Key type ._llcrypt unregistered [ 9865.788754] Key type ._llcrypt registered [ 9865.792729] Key type .llcrypt registered [ 9865.957259] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 9881.991482] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 9883.441161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9883.487410] alg: No test for adler32 (adler32-zlib) [ 9884.650408] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 9885.073887] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 9886.823211] Key type lgssc registered [ 9888.170783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9928.000172] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9928.024186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9929.299476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9929.325597] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9929.419806] Lustre: lustre-MDT0000: new disk, initializing [ 9929.521312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9929.535577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9934.479681] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9946.236099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9946.367681] Lustre: 297511:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9946.434148] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9946.438579] Lustre: Skipped 1 previous similar message [ 9946.536698] Lustre: lustre-MDT0001: new disk, initializing [ 9946.658933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9946.705854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9946.720594] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9951.181636] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9961.995168] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9969.100478] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9976.410980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9976.736588] Lustre: lustre-OST0000: new disk, initializing [ 9976.739801] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9976.770494] Lustre: 299480:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9977.004165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9984.150922] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 9986.111729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9986.123774] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9986.305724] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9995.464112] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10002.571948] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [10014.871291] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [10021.037165] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [10063.332432] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10063.332539] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10063.345845] LustreError: Skipped 1 previous similar message [10063.348100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10063.378555] Lustre: Skipped 1 previous similar message [10064.575535] Lustre: server umount lustre-OST0000 complete [10068.450630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10068.458067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10068.472190] Lustre: Skipped 1 previous similar message [10068.477770] Lustre: Skipped 1 previous similar message [10073.569139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10073.582756] Lustre: Skipped 2 previous similar messages [10074.530737] Lustre: server umount lustre-MDT0000 complete [10077.793426] LustreError: 300337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782251093 with bad export cookie 14140848384361226098 [10077.794359] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10077.800540] LustreError: 300337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10078.159380] Lustre: server umount lustre-MDT0001 complete [10087.024879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10091.038759] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10095.852966] Lustre: server umount MGS complete [10107.814027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10108.432211] LustreError: 303361:0:(qmt_pool.c:1645:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [10108.544966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10113.153165] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10114.019423] LustreError: 303348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10114.047614] LustreError: 303348:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10118.114225] LustreError: 303349:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10122.229800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10122.696783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10127.976920] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10136.137236] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10142.441650] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10149.300984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10149.734224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10156.075544] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10164.126725] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10192.866437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10192.873235] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10192.873792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10192.886967] Lustre: Skipped 1 previous similar message [10197.987321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10198.003572] Lustre: Skipped 1 previous similar message [10205.151478] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10205.306884] Lustre: server umount lustre-OST0000 complete [10209.760371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10209.773480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10209.793825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10209.801891] Lustre: Skipped 3 previous similar messages [10215.466699] Lustre: server umount lustre-MDT0000 complete [10218.465024] LustreError: 303349:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10218.477225] LustreError: 303349:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10219.057498] LustreError: 306075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782251234 with bad export cookie 14140848384361227925 [10219.063367] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10219.068887] LustreError: 306075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10219.463996] Lustre: server umount lustre-MDT0001 complete [10232.071282] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [10234.762300] Key type lgssc unregistered [10235.136559] LNet: 307296:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10235.149711] LNetError: 307296:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10235.166965] LNet: Removed LNI 192.168.204.129@tcp [10236.288104] Key type .llcrypt unregistered [10236.291318] Key type ._llcrypt unregistered [10259.097620] Key type ._llcrypt registered [10259.099767] Key type .llcrypt registered [10259.221791] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [10273.040164] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [10274.891551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10275.114217] alg: No test for adler32 (adler32-zlib) [10276.360299] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [10276.924790] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [10278.663192] Key type lgssc registered [10280.441446] Lustre: Echo OBD driver; http://www.lustre.org/ [10309.265860] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [10325.568835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10325.625127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10326.986591] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10327.082939] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10327.184283] Lustre: lustre-MDT0000: new disk, initializing [10327.260133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10327.278836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10332.428260] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10344.802376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10345.091050] Lustre: lustre-OST0000: new disk, initializing [10345.098370] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10345.103237] Lustre: Skipped 1 previous similar message [10345.110084] Lustre: 311330:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10345.179295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10351.701908] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [10354.757539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10354.765922] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10354.798692] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11327.459553] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11327.486488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11331.403711] Lustre: server umount lustre-MDT0000 complete [11335.621136] LustreError: 310516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782252351 with bad export cookie 5421677045408638519 [11335.626560] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11335.633602] LustreError: 310516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11335.852426] Lustre: server umount lustre-OST0000 complete [11363.798943] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [11376.080589] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [11417.207092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11417.480295] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11417.556696] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11417.687484] Lustre: lustre-MDT0000: new disk, initializing [11417.783862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11417.823610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11423.246239] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11436.736684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11436.887761] Lustre: 315698:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11436.935892] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11436.940158] Lustre: Skipped 1 previous similar message [11437.076768] Lustre: lustre-MDT0001: new disk, initializing [11437.258971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11437.308422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11437.316171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11442.156628] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11453.054211] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11460.061320] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11467.193756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11467.493388] Lustre: lustre-OST0000: new disk, initializing [11467.501380] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11467.511414] Lustre: 317665:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11467.615317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11474.105299] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11477.657512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11477.699175] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11477.987515] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11485.548241] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11491.442554] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11491.673774] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11496.690984] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11496.917537] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11503.587383] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11503.592146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11503.603937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11503.604044] Lustre: Skipped 1 previous similar message [11503.626909] Lustre: Skipped 1 previous similar message [11505.469890] Lustre: server umount lustre-OST0000 complete [11513.826280] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11513.851510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11513.860481] Lustre: Skipped 1 previous similar message [11515.899881] Lustre: server umount lustre-MDT0000 complete [11520.305901] LustreError: 315688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782252536 with bad export cookie 5421677045411579345 [11520.310320] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11520.321310] LustreError: 315688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11520.666180] Lustre: server umount lustre-MDT0001 complete [11524.440229] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [11526.693472] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 18:09:00 (1782252540) [11533.922831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11534.506821] LustreError: 319978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11534.523371] LustreError: 319978:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11534.589092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11538.933101] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11539.942793] LustreError: 319979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11545.058824] LustreError: 319978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11548.478644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11548.768429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11553.720706] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11561.625633] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11567.160882] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11574.349281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11574.641224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11581.003593] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11588.448509] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11595.408902] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11595.624152] Lustre: lustre-OST0001: new disk, initializing [11595.630812] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11595.636544] Lustre: 322815:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11595.712927] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11602.212953] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11604.566626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11604.580593] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11604.622135] LustreError: 323277:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11604.634071] LustreError: 323277:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [11605.673453] LustreError: 323282:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11605.696686] LustreError: 323282:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [11606.694322] LustreError: 323277:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11608.737894] LustreError: 323277:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11608.756242] LustreError: 323277:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [11612.908663] LustreError: 323277:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11612.922624] LustreError: 323277:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [11613.113749] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11613.307491] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11618.216671] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c21b25d4800.ost_server_uuid 50 [11619.802566] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8c21b25d4800.ost_server_uuid in FULL state after 0 sec [11621.153745] LustreError: 323282:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11621.175512] LustreError: 323282:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [11625.860197] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c21b25d4800.ost_server_uuid 50 [11627.929937] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8c21b25d4800.ost_server_uuid in FULL state after 0 sec [11630.532476] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11645.923263] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11645.928206] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11645.932065] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11645.938152] LustreError: Skipped 2 previous similar messages [11645.984404] Lustre: Skipped 2 previous similar messages [11650.580474] Lustre: server umount lustre-OST0001 complete [11651.054866] LustreError: 321800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11651.084243] LustreError: 321800:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11656.176479] LustreError: 324372:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11656.201418] LustreError: 324372:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11656.675735] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11656.686930] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11656.704237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11656.710726] Lustre: Skipped 2 previous similar messages [11658.795531] Lustre: server umount lustre-OST0000 complete [11666.410738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11666.416015] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11666.434330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11666.443310] Lustre: Skipped 1 previous similar message [11666.477218] Lustre: Skipped 1 previous similar message [11669.210370] Lustre: server umount lustre-MDT0000 complete [11671.529112] LustreError: 321572:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11671.574582] LustreError: 321572:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11673.832434] LustreError: 319961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782252689 with bad export cookie 5421677045411580220 [11673.847906] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11683.575956] Lustre: server umount lustre-MDT0001 complete [11694.771782] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [11698.054560] Key type lgssc unregistered [11698.387228] LNet: 325509:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11698.394221] LNetError: 325509:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11698.428544] LNet: Removed LNI 192.168.204.129@tcp [11699.317900] Key type .llcrypt unregistered [11699.321375] Key type ._llcrypt unregistered [11725.130537] Key type ._llcrypt registered [11725.138680] Key type .llcrypt registered [11725.250873] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [11726.460742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11726.529798] alg: No test for adler32 (adler32-zlib) [11727.538774] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [11727.710397] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [11729.375240] Key type lgssc registered [11730.469763] Lustre: Echo OBD driver; http://www.lustre.org/ [11738.060591] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11738.090326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11739.580593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11744.876496] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11749.859365] LustreError: 326928:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11749.879628] LustreError: 326928:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11753.231187] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11754.978578] LustreError: 326929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11760.126240] LustreError: 326929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11760.643607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11761.040695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11766.006542] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11775.590957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11775.917219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11782.555351] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11785.126485] LustreError: 329087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11785.140560] LustreError: 329087:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11785.150813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [11789.669209] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11790.308715] LustreError: 329088:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11790.328847] LustreError: 329088:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11796.048041] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11796.267885] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11801.605098] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [11802.363376] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [11810.071607] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11816.769626] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [11817.028703] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11929.569462] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11929.576921] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11929.588744] LustreError: Skipped 1 previous similar message [11929.609541] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11929.617480] Lustre: Skipped 1 previous similar message [11933.663413] Lustre: server umount lustre-OST0001 complete [11934.694199] LustreError: 329088:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11934.713204] LustreError: 329088:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11939.817647] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11939.827715] Lustre: Skipped 1 previous similar message [11939.835107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11939.846357] Lustre: Skipped 1 previous similar message [11944.930696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11944.939804] Lustre: Skipped 1 previous similar message [11945.404203] Lustre: server umount lustre-OST0000 complete [11950.052452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11950.060357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11950.069048] Lustre: Skipped 2 previous similar messages [11955.170853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11955.178300] Lustre: Skipped 2 previous similar messages [11955.710683] Lustre: server umount lustre-MDT0000 complete [11959.278624] LustreError: 326908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782252975 with bad export cookie 18440416978707645145 [11959.283329] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11959.294684] LustreError: 326908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11959.604725] Lustre: server umount lustre-MDT0001 complete [11971.116822] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [11973.960539] Key type lgssc unregistered [11974.285076] LNet: 331874:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11974.291968] LNetError: 331874:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11974.308148] LNet: Removed LNI 192.168.204.129@tcp [11975.137623] Key type .llcrypt unregistered [11975.141133] Key type ._llcrypt unregistered [12017.398470] Key type ._llcrypt registered [12017.400668] Key type .llcrypt registered [12017.484699] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [12018.199366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12018.217210] alg: No test for adler32 (adler32-zlib) [12019.292549] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [12019.552934] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [12021.279339] Key type lgssc registered [12022.402808] Lustre: Echo OBD driver; http://www.lustre.org/ [12031.297791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12031.326941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12032.670551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12036.340605] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12042.215811] LustreError: 334164:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12042.232400] LustreError: 334164:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12044.320333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12044.559154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12048.812225] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12052.075223] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12059.014787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12059.309488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12065.090441] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12069.556732] LustreError: 335654:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12073.831452] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12074.100924] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12077.217109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [12077.235037] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [12080.908896] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12089.954114] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12099.705618] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [12111.329772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12111.336115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12111.350300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12113.378801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12113.381289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12113.403521] Lustre: Skipped 1 previous similar message [12116.329297] Lustre: server umount lustre-MDT0000 complete [12118.500223] LustreError: 334164:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12118.516618] LustreError: 334164:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12119.680243] LustreError: 335301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782253135 with bad export cookie 16126057330595053142 [12119.687692] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12119.693233] LustreError: 335301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12120.035286] Lustre: server umount lustre-MDT0001 complete [12133.937196] Lustre: server umount lustre-OST0000 complete [12147.926358] Lustre: server umount lustre-OST0001 complete [12165.573382] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [12174.996379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12175.346900] LustreError: 339429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12175.430453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12179.850478] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12180.450857] LustreError: 339430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12188.516530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12188.808312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12192.494703] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12194.858672] Lustre: 340572:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [12200.797855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12206.774307] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12213.428793] LustreError: 340926:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12213.460824] LustreError: 340926:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12215.664375] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12215.898448] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12215.909917] Lustre: Skipped 1 previous similar message [12221.942365] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [12221.947848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [12222.322269] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12231.092848] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12235.155306] Lustre: 342442:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12239.840361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12239.849194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12239.873726] Lustre: Skipped 1 previous similar message [12239.880238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12239.883089] Lustre: Skipped 3 previous similar messages [12242.406825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12242.418243] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12242.434571] Lustre: Skipped 1 previous similar message [12245.530474] Lustre: server umount lustre-MDT0000 complete [12247.526488] LustreError: 340161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12247.562310] LustreError: 340161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12249.690893] LustreError: 339409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782253265 with bad export cookie 16126057330595054843 [12249.697170] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12249.705553] LustreError: 339409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12250.079548] Lustre: server umount lustre-MDT0001 complete [12264.932625] Lustre: server umount lustre-OST0000 complete [12269.023172] Lustre: 333427:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782253268/real 1782253268] req@ffff8cd3c9ae7480 x1868827758541696/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782253284 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12269.043804] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12269.054878] Lustre: Skipped 1 previous similar message [12269.384986] Lustre: server umount lustre-OST0001 complete [12273.695625] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [12275.482420] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [12277.450286] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 18:21:31 (1782253291) [12303.880721] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [12313.642875] loop0: detected capacity change from 0 to 1099511627776 [12314.399072] SGI XFS with ACLs, security attributes, quota, no debug enabled [12314.421538] *** VALIDATE xfs *** [12314.446138] XFS (loop0): Mounting V5 Filesystem [12314.502435] XFS (loop0): Ending clean mount [12315.727080] loop1: detected capacity change from 0 to 18691697672192 [12420.302606] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [12430.668416] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [12468.491414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12468.663520] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12468.685479] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12468.764588] Lustre: lustre-MDT0000: new disk, initializing [12468.843915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12468.875643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12473.313593] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12483.297261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12483.466080] Lustre: 347577:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12483.517454] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12483.521599] Lustre: Skipped 1 previous similar message [12483.589377] Lustre: lustre-MDT0001: new disk, initializing [12483.747111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12483.763026] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12489.548810] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12499.471723] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12505.038748] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12511.674394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12511.911437] Lustre: lustre-OST0000: new disk, initializing [12511.915901] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12511.921915] Lustre: 349547:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12513.477392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12513.488479] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12513.608899] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12518.431723] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12527.988655] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12533.048834] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12533.345618] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12537.866800] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12538.112475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12544.492363] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12544.494165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12544.505527] LustreError: Skipped 1 previous similar message [12544.522238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12544.530523] Lustre: Skipped 4 previous similar messages [12554.207489] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12554.420644] Lustre: server umount lustre-OST0000 complete [12559.843849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12559.864954] Lustre: Skipped 3 previous similar messages [12565.216079] Lustre: server umount lustre-MDT0000 complete [12569.268457] LustreError: 347570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782253585 with bad export cookie 16126057330595056397 [12569.276215] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12569.280348] LustreError: 347570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12569.674064] Lustre: server umount lustre-MDT0001 complete [12579.930833] XFS (loop0): Unmounting Filesystem [12586.369663] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 18:26:40 (1782253600) [12593.738804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12594.105096] LustreError: 352107:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12594.198081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12594.207087] Lustre: Skipped 2 previous similar messages [12599.456927] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12608.031349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12612.436747] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12619.614409] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12624.559102] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12630.788084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12636.641443] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12643.152364] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12654.056110] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12654.063151] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12654.068979] Lustre: Skipped 3 previous similar messages [12654.071901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12654.075270] Lustre: Skipped 2 previous similar messages [12659.119422] Lustre: server umount lustre-OST0000 complete [12664.800606] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12664.811075] LustreError: Skipped 1 previous similar message [12669.595262] Lustre: server umount lustre-MDT0000 complete [12670.438058] LustreError: 352107:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12670.455721] LustreError: 352107:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12673.530127] LustreError: 352087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782253689 with bad export cookie 16126057330595057279 [12673.531734] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12673.538312] LustreError: 352087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12673.838757] Lustre: server umount lustre-MDT0001 complete [12683.734558] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [12686.176884] Key type lgssc unregistered [12686.451444] LNet: 355737:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12686.458722] LNetError: 355737:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12686.474323] LNet: Removed LNI 192.168.204.129@tcp [12687.304227] Key type .llcrypt unregistered [12687.307731] Key type ._llcrypt unregistered [12716.509099] Key type ._llcrypt registered [12716.510844] Key type .llcrypt registered [12716.701104] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [12717.919771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12717.962936] alg: No test for adler32 (adler32-zlib) [12719.211463] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [12719.452289] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [12721.273106] Key type lgssc registered [12722.958994] Lustre: Echo OBD driver; http://www.lustre.org/ [12731.782984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12731.807926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12733.240730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12737.494449] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12743.652397] LustreError: 357305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12743.667181] LustreError: 357305:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12747.214385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12747.648965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12753.742889] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12762.720695] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12768.591919] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12775.945220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12776.477292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12783.236492] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12790.785298] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12803.033204] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.29@tcp (stopping) [12803.041943] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12807.865386] Lustre: server umount lustre-MDT0001 complete [12808.124944] LustreError: 358613:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12813.248338] LustreError: 357302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12813.268928] LustreError: 357302:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12818.360560] LustreError: 357302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12818.381217] LustreError: 357302:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12819.015497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12819.153587] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12819.157848] Lustre: Found index 1 for lustre-MDT0001, updating log [12819.200868] Lustre: 357294:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12819.560710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12824.194420] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [12825.080177] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12828.157882] LustreError: 360908:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12828.170893] LustreError: 360908:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12828.185228] LustreError: 357297:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12828.290000] LustreError: 360911:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.129@tcp: cfg command failed: rc = -17 [12828.303047] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [12828.303047] [13132.256795] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13132.260117] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13132.278625] Lustre: Skipped 2 previous similar messages [13132.297746] Lustre: Skipped 3 previous similar messages [13135.841675] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.29@tcp (stopping) [13137.172971] Lustre: server umount lustre-MDT0001 complete [13137.382791] LustreError: 357324:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13137.415347] LustreError: 357324:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [13146.053085] LustreError: 359235:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13146.099443] LustreError: 359235:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [13150.971293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13151.089736] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13151.095907] Lustre: Found index 1 for lustre-MDT0001, updating log [13151.133511] Lustre: 357294:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13151.445888] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13156.310750] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [13156.859356] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13158.894316] LustreError: 369451:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13158.907126] LustreError: 369451:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13158.925778] LustreError: 357297:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13158.936682] LustreError: 357297:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13159.059718] LustreError: 369456:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [13159.065404] LustreError: 369456:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [13159.073812] LustreError: 369456:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.129@tcp: cfg command failed: rc = -17 [13159.087206] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.204.129@tcp [13159.087206] [13463.513592] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.29@tcp (stopping) [13464.043909] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13468.599724] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.29@tcp (stopping) [13468.612651] Lustre: Skipped 2 previous similar messages [13468.882587] Lustre: server umount lustre-MDT0001 complete [13469.155270] LustreError: 357305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13469.170063] LustreError: 357305:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13479.330391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13479.431404] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13479.438410] Lustre: Found index 1 for lustre-MDT0001, updating log [13479.493840] Lustre: 357293:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13479.819149] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13484.523438] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [13485.053129] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13488.126913] LustreError: 377857:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13488.144392] LustreError: 377857:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13488.177357] LustreError: 357297:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13488.194130] LustreError: 357297:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13488.269073] LustreError: 377860:0:(lod_lov.c:212:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [13488.294974] LustreError: 377860:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.129@tcp: cfg command failed: rc = -114 [13488.306045] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [13488.306045] [13807.589740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13807.596033] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13807.612762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13807.642288] Lustre: Skipped 2 previous similar messages [13811.688118] Lustre: server umount lustre-MDT0000 complete [13812.706719] LustreError: 358613:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13812.740147] LustreError: 358613:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [13816.050066] LustreError: 357287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782254831 with bad export cookie 7243069909066777631 [13816.058463] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13816.067681] LustreError: 357287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13817.826261] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13817.839512] Lustre: Skipped 1 previous similar message [13822.724435] Lustre: server umount lustre-MDT0001 complete [13826.722738] Lustre: server umount lustre-OST0000 complete [13836.615828] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [13846.414502] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [13884.241315] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 18:48:18 (1782254898) [13892.163680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13892.364675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13892.394323] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13892.504384] Lustre: lustre-MDT0000: new disk, initializing [13892.617547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13892.666695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13897.849720] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [13910.129025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13910.274868] Lustre: 388436:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13910.331337] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13910.339049] Lustre: Skipped 1 previous similar message [13910.478982] Lustre: lustre-MDT0001: new disk, initializing [13910.638857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13910.695377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13910.719074] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13916.099763] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [13926.335194] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13932.825478] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13939.824588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13940.071372] Lustre: lustre-OST0000: new disk, initializing [13940.075329] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13940.080731] Lustre: 390406:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13940.167551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13946.636710] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [13946.992933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13947.022439] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13947.201720] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13957.014932] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13965.795493] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13965.804628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13965.818605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13965.826884] Lustre: Skipped 4 previous similar messages [13967.303524] Lustre: server umount lustre-MDT0000 complete [13967.841832] LustreError: 389814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13967.860332] LustreError: 389814:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13971.019395] LustreError: 390405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782254986 with bad export cookie 7243069909066779584 [13971.026781] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13971.032877] LustreError: 390405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13971.046104] LustreError: 389242:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [13971.335669] Lustre: server umount lustre-MDT0001 complete [13985.462941] Lustre: server umount lustre-OST0000 complete [14014.722964] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [14025.587549] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [14058.038739] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13787 sec ========= 18:51:12 (1782255072) [14059.770549] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:51:14 (1782255074) === [14063.441378] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:51:17 (1782255077) === [14105.356882] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [14108.793420] Key type lgssc unregistered [14109.201587] LNet: 396770:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14109.209986] LNetError: 396770:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14109.256767] LNet: Removed LNI 192.168.204.129@tcp [14110.390168] Key type .llcrypt unregistered [14110.391943] Key type ._llcrypt unregistered