[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-22) (GCC)) #7 SMP Sat Jan 18 21:01:29 EST 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-1.fc39 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 528244816 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcbe3000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 513 pages/cpu s2064384 r8192 d28672 u4194304 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2818960K/4306352K available (20483K kernel code, 12066K rwdata, 7356K rodata, 4680K init, 23504K bss, 542472K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41388 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x616/0x99a with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar [ 0.001000] ... MAX_LOCKDEP_SUBCLASSES: 8 [ 0.001000] ... MAX_LOCK_DEPTH: 48 [ 0.001000] ... MAX_LOCKDEP_KEYS: 8192 [ 0.001000] ... CLASSHASH_SIZE: 4096 [ 0.001000] ... MAX_LOCKDEP_ENTRIES: 32768 [ 0.001000] ... MAX_LOCKDEP_CHAINS: 65536 [ 0.001000] ... CHAINHASH_SIZE: 32768 [ 0.001000] memory used by lock dependency info: 4149 kB [ 0.001000] per task-struct memory footprint: 2688 bytes [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001018] APIC: Switch to symmetric I/O mode setup [ 0.003361] x2apic enabled [ 0.004016] Switched APIC routing to physical x2apic. [ 0.005021] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008049] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009018] pid_max: default: 32768 minimum: 301 [ 0.010397] LSM: Security Framework initializing [ 0.012140] Yama: becoming mindful. [ 0.013164] SELinux: Initializing. [ 0.014311] *** VALIDATE selinux *** [ 0.023318] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029653] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031171] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032154] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033315] *** VALIDATE tmpfs *** [ 0.036358] *** VALIDATE proc *** [ 0.038614] *** VALIDATE cgroup *** [ 0.039015] *** VALIDATE cgroup2 *** [ 0.040395] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041244] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042015] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043047] Spectre V2 : User space: Vulnerable [ 0.044016] Speculative Store Bypass: Vulnerable [ 0.046932] debug: unmapping init [mem 0xffffffffa8303000-0xffffffffa830afff] [ 0.048955] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.051111] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.052050] ... version: 2 [ 0.053020] ... bit width: 48 [ 0.054019] ... generic registers: 4 [ 0.055020] ... value mask: 0000ffffffffffff [ 0.056022] ... max period: 00007fffffffffff [ 0.057018] ... fixed-purpose events: 3 [ 0.058015] ... event mask: 000000070000000f [ 0.059579] rcu: Hierarchical SRCU implementation. [ 0.063975] smp: Bringing up secondary CPUs ... [ 0.065596] x86: Booting SMP configuration: [ 0.066089] .... node #0, CPUs: #1 [ 0.070711] #2 [ 0.074731] #3 [ 0.076541] smp: Brought up 1 node, 4 CPUs [ 0.077013] smpboot: Max logical packages: 1 [ 0.078030] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.132101] node 0 deferred pages initialised in 50ms [ 0.133000] pgdatinit0 (35) used greatest stack depth: 14528 bytes left [ 0.139427] devtmpfs: initialized [ 0.141286] x86/mm: Memory block size: 128MB [ 0.154251] gcov: version magic: 0x41383552 [ 0.159924] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.160157] futex hash table entries: 1024 (order: 5, 131072 bytes, vmalloc) [ 0.161741] pinctrl core: initialized pinctrl subsystem [ 0.163064] [ 0.163627] ************************************************************* [ 0.164019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165017] ** ** [ 0.166020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.167018] ** ** [ 0.168019] ** This means that this kernel is built to expose internal ** [ 0.169018] ** IOMMU data structures, which may compromise security on ** [ 0.170017] ** your system. ** [ 0.171018] ** ** [ 0.172022] ** If you see this message and you are not debugging the ** [ 0.173019] ** kernel, report this immediately to your vendor! ** [ 0.174022] ** ** [ 0.175031] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.176023] ************************************************************* [ 0.179012] NET: Registered protocol family 16 [ 0.182120] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.183110] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.184127] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.187338] cpuidle: using governor menu [ 0.188979] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.192194] PCI: Using configuration type 1 for base access [ 0.194127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.242133] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.244372] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.259068] cryptd: max_cpu_qlen set to 1000 [ 0.263864] ACPI: Added _OSI(Module Device) [ 0.265018] ACPI: Added _OSI(Processor Device) [ 0.267017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.268016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.310120] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.324478] ACPI: Interpreter enabled [ 0.326256] ACPI: PM: (supports S0 S3 S4 S5) [ 0.328022] ACPI: Using IOAPIC for interrupt routing [ 0.330310] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.337914] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.421241] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.423078] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.426021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.429342] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.441115] acpiphp: Slot [2] registered [ 0.443353] acpiphp: Slot [5] registered [ 0.444345] acpiphp: Slot [6] registered [ 0.446316] acpiphp: Slot [7] registered [ 0.447348] acpiphp: Slot [8] registered [ 0.449302] acpiphp: Slot [9] registered [ 0.451312] acpiphp: Slot [10] registered [ 0.452326] acpiphp: Slot [3] registered [ 0.454325] acpiphp: Slot [4] registered [ 0.456297] acpiphp: Slot [11] registered [ 0.458156] acpiphp: Slot [12] registered [ 0.459316] acpiphp: Slot [13] registered [ 0.460332] acpiphp: Slot [14] registered [ 0.461352] acpiphp: Slot [15] registered [ 0.462313] acpiphp: Slot [16] registered [ 0.463303] acpiphp: Slot [17] registered [ 0.464300] acpiphp: Slot [18] registered [ 0.465384] acpiphp: Slot [19] registered [ 0.466309] acpiphp: Slot [20] registered [ 0.467324] acpiphp: Slot [21] registered [ 0.468371] acpiphp: Slot [22] registered [ 0.469332] acpiphp: Slot [23] registered [ 0.470349] acpiphp: Slot [24] registered [ 0.471358] acpiphp: Slot [25] registered [ 0.472362] acpiphp: Slot [26] registered [ 0.473320] acpiphp: Slot [27] registered [ 0.474330] acpiphp: Slot [28] registered [ 0.475326] acpiphp: Slot [29] registered [ 0.476319] acpiphp: Slot [30] registered [ 0.477384] acpiphp: Slot [31] registered [ 0.478182] PCI host bridge to bus 0000:00 [ 0.479039] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.480042] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.481048] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.482049] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.483067] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.484046] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.485392] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.488271] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.491598] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.497018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.500773] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.501022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.502025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.503024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.505864] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.507018] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.508053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.510995] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.513834] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.520025] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.524018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.542922] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.545028] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.548021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.555020] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.575791] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.578017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.581017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.588017] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.611039] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.620032] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.627024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.644025] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.665713] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.668021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.671016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.679023] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.701124] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.704020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.707016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.714019] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.734709] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.737018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.740019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.748019] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.774449] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.776878] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.778922] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.780871] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.781959] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.790067] iommu: Default domain type: Passthrough [ 0.791239] SCSI subsystem initialized [ 0.793327] ACPI: bus type USB registered [ 0.795431] usbcore: registered new interface driver usbfs [ 0.796253] usbcore: registered new interface driver hub [ 0.798170] usbcore: registered new device driver usb [ 0.800723] pps_core: LinuxPPS API ver. 1 registered [ 0.802013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.805190] PTP clock support registered [ 0.812123] EDAC MC: Ver: 3.0.0 [ 0.816109] PCI: Using ACPI for IRQ routing [ 0.819642] NetLabel: Initializing [ 0.820016] NetLabel: domain hash size = 128 [ 0.821016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.822365] NetLabel: unlabeled traffic allowed by default [ 0.823613] vgaarb: loaded [ 0.824513] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.826032] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.834344] clocksource: Switched to clocksource kvm-clock [ 1.280187] VFS: Disk quotas dquot_6.6.0 [ 1.281858] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.284410] *** VALIDATE ramfs *** [ 1.285758] *** VALIDATE hugetlbfs *** [ 1.287878] pnp: PnP ACPI init [ 1.297235] pnp: PnP ACPI: found 6 devices [ 1.343711] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.347740] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.349968] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.352604] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.355690] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.358318] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 1.362540] NET: Registered protocol family 2 [ 1.365931] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.372786] tcp_listen_portaddr_hash hash table entries: 4096 (order: 6, 360448 bytes, vmalloc) [ 1.376643] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.383659] TCP bind hash table entries: 65536 (order: 10, 5242880 bytes, vmalloc) [ 1.390952] TCP: Hash tables configured (established 65536 bind 65536) [ 1.395512] MPTCP token hash table entries: 8192 (order: 7, 786432 bytes, vmalloc) [ 1.399407] UDP hash table entries: 4096 (order: 7, 786432 bytes, vmalloc) [ 1.403436] UDP-Lite hash table entries: 4096 (order: 7, 786432 bytes, vmalloc) [ 1.407095] NET: Registered protocol family 1 [ 1.412146] RPC: Registered named UNIX socket transport module. [ 1.414301] RPC: Registered udp transport module. [ 1.415904] RPC: Registered tcp transport module. [ 1.417552] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.420237] NET: Registered protocol family 44 [ 1.421586] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.423666] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.425928] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.428406] PCI: CLS 0 bytes, default 64 [ 1.430798] Unpacking initramfs... [ 3.511858] debug: unmapping init [mem 0xffff9cdf7cbe3000-0xffff9cdf7ffbffff] [ 3.515615] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.517777] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.520316] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.532491] cryptomgr_test (65) used greatest stack depth: 14248 bytes left [ 3.540778] cryptomgr_test (67) used greatest stack depth: 14008 bytes left [ 4.389412] Initialise system trusted keyrings [ 4.391199] Key type blacklist registered [ 4.393548] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.456749] zbud: loaded [ 4.476503] *** VALIDATE nfs *** [ 4.477541] *** VALIDATE nfs4 *** [ 4.484822] pstore: using deflate compression [ 4.491370] Platform Keyring initialized [ 4.515381] cryptomgr_test (84) used greatest stack depth: 13976 bytes left [ 4.527716] cryptomgr_test (86) used greatest stack depth: 13800 bytes left [ 4.564739] cryptomgr_test (94) used greatest stack depth: 13640 bytes left [ 4.698766] NET: Registered protocol family 38 [ 4.700555] Key type asymmetric registered [ 4.702061] Asymmetric key parser 'x509' registered [ 4.704171] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.707684] io scheduler mq-deadline registered [ 4.709931] io scheduler kyber registered [ 4.713355] io scheduler bfq registered [ 4.715123] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.720580] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.725139] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.728739] ACPI: Power Button [PWRF] [ 6.101699] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 7.501790] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 10.250109] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 11.635245] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 14.397853] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 14.436485] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 14.466579] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 14.475976] Non-volatile memory driver v1.3 [ 14.478370] Linux agpgart interface v0.103 [ 14.590623] virtio_blk virtio1: [vda] 131896 512-byte logical blocks (67.5 MB/64.4 MiB) [ 14.593694] vda: detected capacity change from 0 to 67530752 [ 14.622070] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 14.625380] vdb: detected capacity change from 0 to 1073741824 [ 14.649277] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 14.652557] vdc: detected capacity change from 0 to 2621440000 [ 14.684964] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 14.688310] vdd: detected capacity change from 0 to 2621440000 [ 14.712203] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 14.715271] vde: detected capacity change from 0 to 4294967296 [ 14.737493] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 14.740477] vdf: detected capacity change from 0 to 4294967296 [ 14.755661] libphy: Fixed MDIO Bus: probed [ 14.768938] usbcore: registered new interface driver usbserial_generic [ 14.771241] usbserial: USB Serial support registered for generic [ 14.773573] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 14.778112] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 14.780153] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 14.784197] mousedev: PS/2 mouse device common for all mice [ 14.789371] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 14.793036] rtc_cmos 00:05: RTC can wake from S4 [ 14.798135] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 14.798708] rtc_cmos 00:05: registered as rtc0 [ 14.802836] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 14.803344] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 14.804812] intel_pstate: CPU model not supported [ 14.812938] hid: raw HID events driver (C) Jiri Kosina [ 14.815874] usbcore: registered new interface driver usbhid [ 14.817851] usbhid: USB HID core driver [ 14.819600] drop_monitor: Initializing network drop monitor service [ 14.821442] Initializing XFRM netlink socket [ 14.825029] NET: Registered protocol family 10 [ 14.831875] Segment Routing with IPv6 [ 14.832978] NET: Registered protocol family 17 [ 14.834622] mpls_gso: MPLS GSO support [ 14.842763] RAS: Correctable Errors collector initialized. [ 14.844929] AVX version of gcm_enc/dec engaged. [ 14.846513] AES CTR mode by8 optimization enabled [ 15.031471] sched_clock: Marking stable (15031382976, 0)->(16251105684, -1219722708) [ 15.038672] registered taskstats version 1 [ 15.044400] Loading compiled-in X.509 certificates [ 15.047512] zswap: loaded using pool lzo/zbud [ 15.109298] Key type big_key registered [ 15.135160] Key type encrypted registered [ 15.136931] ima: No TPM chip found, activating TPM-bypass! [ 15.138683] ima: Allocated hash algorithm: sha1 [ 15.140268] ima: No architecture policies found [ 15.142076] evm: Initialising EVM extended attributes: [ 15.143636] evm: security.selinux [ 15.144857] evm: security.ima [ 15.145860] evm: security.capability [ 15.147090] evm: HMAC attrs: 0x1 [ 15.155995] rtc_cmos 00:05: setting system clock to 2025-04-01 08:24:30 UTC (1743495870) [ 15.187254] debug: unmapping init [mem 0xffffffffa9803000-0xffffffffa99fffff] [ 15.190558] debug: unmapping init [mem 0xffffffffa7e71000-0xffffffffa8302fff] [ 15.197234] Write protecting the kernel read-only data: 30720k [ 15.201297] debug: unmapping init [mem 0xffffffffa6403000-0xffffffffa65fffff] [ 15.204607] debug: unmapping init [mem 0xffffffffa6d2f000-0xffffffffa6dfffff] [ 15.320496] systemd[1]: systemd 239 (239-82.el8_10.3) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.329627] systemd[1]: Detected virtualization kvm. [ 15.331528] systemd[1]: Detected architecture x86-64. [ 15.333516] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 15.373686] systemd[1]: No hostname configured. [ 15.375541] systemd[1]: Set hostname to . [ 15.379033] random: systemd: uninitialized urandom read (16 bytes read) [ 15.382701] systemd[1]: Initializing machine ID from random generator. [ 15.802142] random: systemd: uninitialized urandom read (16 bytes read) [ 15.805617] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 15.810769] random: systemd: uninitialized urandom read (16 bytes read) [ 15.813080] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 15.818608] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... [ 16.040216] memstrack-start (230) used greatest stack depth: 13528 bytes left Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 17.493268] device-mapper: uevent: version 1.0.3 [ 17.496791] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ 18.636870] udevadm (416) used greatest stack depth: 13392 bytes left Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 19.957901] random: fast init done Starting dracut initqueue hook... [ 20.216873] virtio_net virtio0 ens2: renamed from eth0 [ 20.387445] scsi host0: ata_piix [ 20.449954] scsi host1: ata_piix [ 20.487304] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 20.489714] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 21.222527] systemd-udevd (453) used greatest stack depth: 12504 bytes left [ 21.936442] ip (528) used greatest stack depth: 11496 bytes left [ 24.660766] random: crng init done [ 24.662372] random: 7 urandom warning(s) missed due to ratelimiting [ 26.127674] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 27.718370] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 33.683469] printk: systemd: 25 output lines suppressed due to ratelimiting [ 34.811335] SELinux: Disabled at runtime. [ 35.051799] systemd[1]: systemd 239 (239-82.el8_10.3) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 35.072705] systemd[1]: Detected virtualization kvm. [ 35.074479] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 37.980348] systemd[1]: initrd-switch-root.service: Succeeded. [ 38.021493] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 38.037910] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 38.042235] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 38.058830] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 38.109099] systemd[1]: Starting Journal Service... Starting Journal Service... [ 38.126423] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. [ 38.717564] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Mounted /mnt. [ 41.298078] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ OK ] Started udev Coldplug all Devices. [ 43.730956] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 44.885111] input: PC Speaker as /devices/platform/pcspkr/input/input5 [* ] A start job is running for Configur…only root support (10s / no limit)[ 48.758533] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [** ] A start job is running for Configur…only root support (11s / no limit)[ 49.213452] EDAC sbridge: Ver: 1.1.2 [*** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ *] A start job is running for Configur…only root support (14s / no limit) [ **] A start job is running for Configur…only root support (14s / no limit) [ ***] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (15s / no limit) [ *** ] A start job is running for Configur…only root support (16s / no limit) [*** ] A start job is running for Configur…only root support (16s / no limit) [** ] A start job is running for Configur…only root support (16s / no limit) [* ] A start job is running for Configur…only root support (17s / no limit) [** ] A start job is running for Configur…only root support (17s / no limit) [*** ] A start job is running for Configur…only root support (18s / no limit)[ 57.002911] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (19s / no limit) [ *** ] A start job is running for Configur…only root support (19s / no limit)[ 58.013815] NFS: Registering the id_resolver key type [ 58.026785] Key type id_resolver registered [ 58.037053] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (20s / no limit) [ **] A start job is running for Configur…only root support (20s / no limit) [ *] A start job is running for Configur…only root support (21s / no limit)[ 58.987233] mount.nfs (975) used greatest stack depth: 10760 bytes left [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ 63.576101] hrtimer: interrupt took 3151772 ns [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg607-server login: [ 173.095394] libcfs: loading out-of-tree module taints kernel. [ 173.189186] Key type ._llcrypt registered [ 173.193238] Key type .llcrypt registered [ 173.517358] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 207.109578] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 212.811365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 212.890667] alg: No test for adler32 (adler32-zlib) [ 215.064372] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 216.609774] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 216.615743] LNet: Accept secure, port 988 [ 218.560384] Key type lgssc registered [ 222.124512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 257.557591] ZFS: Loaded module v2.3.0-1, ZFS pool version 5000, ZFS filesystem version 5 [ 257.566028] modprobe (4341) used greatest stack depth: 5584 bytes left [ 265.860830] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 287.042618] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 303.314238] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 318.316555] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 357.616933] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 391.845338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 392.070587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 392.134673] ------------[ cut here ]------------ [ 392.137772] DEBUG_LOCKS_WARN_ON(!lockdep_enabled()) [ 392.137802] WARNING: CPU: 0 PID: 6527 at kernel/locking/lockdep.c:4713 lockdep_init_map_type+0x29d/0x410 [ 392.149385] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 392.208943] CPU: 0 PID: 6527 Comm: mount.lustre Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #7 [ 392.220467] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-1.fc39 04/01/2014 [ 392.246458] RIP: 0010:lockdep_init_map_type+0x29d/0x410 [ 392.255814] Code: c0 0f 85 db fe ff ff 48 c7 c6 66 64 b9 a6 48 c7 c7 a7 4e b7 a6 48 83 05 f0 df 25 03 01 e8 27 14 f5 ff 48 83 05 eb df 25 03 01 <0f> 0b 48 83 05 e9 df 25 03 01 48 83 05 e9 df 25 03 01 e9 a1 fe ff [ 392.299918] RSP: 0018:ffffb552c239f748 EFLAGS: 00010202 [ 392.312534] RAX: 0000000000000000 RBX: ffff9cdffe1701f8 RCX: 0000000000000001 [ 392.328601] RDX: 0000000000000001 RSI: 00000000ffff7fff RDI: ffff9ce0013de800 [ 392.337148] RBP: ffffffffc14f67c0 R08: 0000000000000000 R09: c0000000ffff7fff [ 392.354924] R10: 0000000000000001 R11: ffffb552c239f538 R12: 0000000000000002 [ 392.371773] R13: ffff9cdfcfaec000 R14: 0000000000000000 R15: 0000000000000001 [ 392.374128] FS: 00007f6f670deb40(0000) GS:ffff9ce001200000(0000) knlGS:0000000000000000 [ 392.377456] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 392.379823] CR2: 0000556fba864078 CR3: 000000010fa0f002 CR4: 0000000000170ef0 [ 392.382097] Call Trace: [ 392.383202] ? show_regs.cold.9+0x22/0x2f [ 392.385354] ? __warn+0xc8/0x150 [ 392.386413] ? lockdep_init_map_type+0x29d/0x410 [ 392.388084] ? report_bug+0x113/0x140 [ 392.389103] ? do_error_trap+0xb6/0x130 [ 392.390189] ? do_invalid_op+0x46/0x60 [ 392.398721] ? lockdep_init_map_type+0x29d/0x410 [ 392.405202] ? invalid_op+0x14/0x20 [ 392.406423] ? lockdep_init_map_type+0x29d/0x410 [ 392.411927] ? lockdep_init_map_type+0x295/0x410 [ 392.413558] ldiskfs_enable_quotas+0x1b9/0x4a0 [ldiskfs] [ 392.417374] ldiskfs_fill_super+0x3a56/0x43c0 [ldiskfs] [ 392.428174] ? ldiskfs_calculate_overhead+0x670/0x670 [ldiskfs] [ 392.429812] ? mount_bdev+0x226/0x270 [ 392.430632] mount_bdev+0x226/0x270 [ 392.438420] ldiskfs_mount+0x19/0x30 [ldiskfs] [ 392.443920] legacy_get_tree+0x35/0x90 [ 392.448792] vfs_get_tree+0x2a/0x140 [ 392.452378] fc_mount+0x16/0x60 [ 392.458359] vfs_kern_mount+0x91/0x100 [ 392.462059] osd_mount+0x5c4/0x1080 [osd_ldiskfs] [ 392.466096] osd_device_init0+0x2e1/0xc20 [osd_ldiskfs] [ 392.469054] osd_device_alloc+0x22a/0x290 [osd_ldiskfs] [ 392.477698] obd_setup+0x196/0x430 [obdclass] [ 392.482204] class_setup+0x6f5/0x9f0 [obdclass] [ 392.489182] class_process_config+0x1658/0x2b60 [obdclass] [ 392.497579] do_lcfg+0x376/0x740 [obdclass] [ 392.502948] lustre_start_simple+0x8f/0x220 [obdclass] [ 392.505110] osd_start+0x6aa/0xb60 [ptlrpc] [ 392.513785] ? server_name2index+0x79/0xe0 [obdclass] [ 392.518729] ? lsi_prepare+0x2e7/0x690 [ptlrpc] [ 392.524569] server_fill_super+0x99/0x1190 [ptlrpc] [ 392.528920] ? obd_zombie_barrier+0x63/0x120 [obdclass] [ 392.535424] ? debug_mutex_init+0x43/0x60 [ 392.540192] lustre_fill_super+0x4a6/0x5e0 [lustre] [ 392.545233] ? lustre_mount+0x30/0x30 [lustre] [ 392.552060] mount_nodev+0x56/0xf0 [ 392.553331] lustre_mount+0x1c/0x30 [lustre] [ 392.556518] legacy_get_tree+0x35/0x90 [ 392.557666] vfs_get_tree+0x2a/0x140 [ 392.558708] do_mount+0xd84/0x1190 [ 392.565136] ksys_mount+0x11d/0x150 [ 392.566194] __x64_sys_mount+0x29/0x40 [ 392.567405] do_syscall_64+0xc1/0x450 [ 392.573422] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 392.578333] RIP: 0033:0x7f6f63212dbe [ 392.584752] Code: 48 8b 0d cd 60 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 a5 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 9a 60 39 00 f7 d8 64 89 01 48 [ 392.599439] RSP: 002b:00007ffedce7bb38 EFLAGS: 00000286 ORIG_RAX: 00000000000000a5 [ 392.607268] RAX: ffffffffffffffda RBX: 0000000000430cf6 RCX: 00007f6f63212dbe [ 392.615179] RDX: 0000000000430cf6 RSI: 00007ffedce821e0 RDI: 0000000001776940 [ 392.621974] RBP: 00007ffedce821e0 R08: 0000000001776960 R09: 0000000001776010 [ 392.638334] R10: 0000000001000000 R11: 0000000000000286 R12: 0000000000000000 [ 392.644760] R13: 0000000000654920 R14: 00000000fffffff5 R15: 00000000fffffff5 [ 392.666359] ---[ end trace 1e9d4f19d8c8a367 ]--- [ 392.671835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 394.334974] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 394.436212] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 394.549209] Lustre: lustre-MDT0000: new disk, initializing [ 394.733947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 394.789968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 401.324155] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 424.054544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 424.198210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 424.384376] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 424.420304] Lustre: 7461:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 424.477739] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 424.484155] Lustre: Skipped 1 previous similar message [ 424.860520] Lustre: lustre-MDT0001: new disk, initializing [ 425.058783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 425.178235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 425.184380] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 432.157088] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 440.325227] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 456.728664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 456.875213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 457.460328] Lustre: 8421:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 457.499685] Lustre: lustre-OST0000: new disk, initializing [ 457.509833] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 457.666441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 465.002305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 465.030770] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 465.270877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 466.871478] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 489.019130] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 489.165462] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 489.347382] Lustre: 9437:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 489.384646] Lustre: lustre-OST0001: new disk, initializing [ 489.396399] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 489.489573] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 496.234420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 496.239381] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 496.490582] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 498.602958] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 516.091033] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 527.294165] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 544.633210] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing check_logdir /tmp/testlogs/ [ 553.735159] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing yml_node [ 570.117910] Lustre: DEBUG MARKER: Client: 2.16.52.73 [ 576.016332] Lustre: DEBUG MARKER: MDS: 2.16.52.73 [ 582.289760] Lustre: DEBUG MARKER: OSS: 2.16.52.73 [ 586.006877] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Apr 1 04:33:59 EDT 2025 [ 616.169599] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 619.807520] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 639.971246] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 639.976409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 639.977429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 640.017166] Lustre: Skipped 2 previous similar messages [ 645.091372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 645.114614] Lustre: Skipped 6 previous similar messages [ 646.242341] Lustre: server umount lustre-MDT0000 complete [ 650.215397] Lustre: lustre-MDT0001: haven't heard from client 7f280239-f01d-459f-a4bd-e2e3139a51cb (at 192.168.206.7@tcp) in 31 seconds. I think it's dead, and I am evicting it. exp ffff9cded4a50800, cur 1743496505 expire 1743496475 last 1743496474 [ 652.741363] LustreError: 6538:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743496508 with bad export cookie 5624132132395731816 [ 652.745330] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 652.750974] LustreError: 6538:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 655.329494] LustreError: 6551:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 655.337160] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 655.339359] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 655.358937] LustreError: 6551:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 6 previous similar messages [ 655.417725] Lustre: Skipped 2 previous similar messages [ 657.385067] LustreError: 6552:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 657.385125] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 657.417166] LustreError: 6552:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 657.423726] Lustre: Skipped 2 previous similar messages [ 659.513223] Lustre: server umount lustre-MDT0001 complete [ 666.678780] Lustre: server umount lustre-OST0000 complete [ 673.341744] Lustre: server umount lustre-OST0001 complete [ 685.372418] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 706.409745] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 728.173972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 741.903065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 753.624309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 766.747139] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 777.448669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 777.556332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 778.199585] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 778.266059] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 778.402550] Lustre: lustre-MDT0000: new disk, initializing [ 778.594983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 778.638868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 784.780680] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 802.320633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 802.522450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 802.861545] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 802.911817] Lustre: 15722:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 803.014076] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 803.026450] Lustre: Skipped 1 previous similar message [ 803.203386] Lustre: lustre-MDT0001: new disk, initializing [ 803.569559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 803.650898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 803.660387] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 810.114680] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 822.036360] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 830.236853] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 841.844446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 842.046638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 842.736509] Lustre: 16673:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 842.844645] Lustre: lustre-OST0000: new disk, initializing [ 842.849670] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 843.060697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 843.949468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 843.964140] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 844.156079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 852.065530] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 866.011983] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 874.096524] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 874.622195] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 881.113214] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 881.545987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 885.222957] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 885.223586] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 885.225850] LustreError: Skipped 1 previous similar message [ 885.230168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 885.252190] Lustre: Skipped 1 previous similar message [ 885.273910] Lustre: Skipped 1 previous similar message [ 891.508081] Lustre: server umount lustre-OST0000 complete [ 900.591757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 900.597857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 900.613356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 900.634981] Lustre: Skipped 2 previous similar messages [ 905.007727] Lustre: server umount lustre-MDT0000 complete [ 905.702698] LustreError: 14970:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 905.722364] LustreError: 14970:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 910.817700] LustreError: 14970:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 910.915311] LustreError: 14970:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 912.484964] LustreError: 14950:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743496767 with bad export cookie 5624132132395733349 [ 912.486477] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 912.498953] LustreError: 14950:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 913.359333] Lustre: server umount lustre-MDT0001 complete [ 923.973038] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 04:39:37 (1743496777) [ 937.444456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 938.139532] LustreError: 18809:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 938.328844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 945.856888] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 947.696648] LustreError: 18808:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 947.717453] LustreError: 18808:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 961.257411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 961.429685] Lustre: 19506:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 968.863961] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 980.410036] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 987.417355] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 999.630115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 999.919317] Lustre: 20390:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1010.226359] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1020.914617] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1028.220562] Lustre: Modifying parameter general.osc.*.max_dirty_mb in log params [ 1038.818202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1038.826357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1038.849533] Lustre: Skipped 1 previous similar message [ 1038.865610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1040.253099] Lustre: server umount lustre-MDT0000 complete [ 1044.454214] LustreError: 18809:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1044.489830] LustreError: 18809:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 4 previous similar messages [ 1046.262134] LustreError: 18790:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743496901 with bad export cookie 5624132132395734231 [ 1046.262134] LustreError: 18789:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743496901 with bad export cookie 5624132132395734231 [ 1046.262816] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1046.283762] LustreError: 18790:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1046.939935] Lustre: server umount lustre-MDT0001 complete [ 1064.270393] Lustre: server umount lustre-OST0000 complete [ 1109.016522] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 1129.577256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1130.346931] LustreError: 22843:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1130.460129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1130.472205] Lustre: Skipped 2 previous similar messages [ 1136.910383] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1152.655945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1152.908466] Lustre: 23543:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1160.527654] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1165.722889] Lustre: Setting parameter general.debug_raw_pointers in log params [ 1176.226685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1187.174592] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1202.051614] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1202.275646] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1202.909897] Lustre: lustre-OST0001: new disk, initializing [ 1202.921661] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1204.981988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1204.996433] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1205.286300] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1213.567861] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1230.133751] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1237.243220] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 1261.541461] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1261.544811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1261.577552] Lustre: Skipped 3 previous similar messages [ 1261.594844] Lustre: Skipped 5 previous similar messages [ 1263.435838] Lustre: server umount lustre-MDT0000 complete [ 1266.154382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1266.158339] LustreError: 23552:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1266.164748] LustreError: 23552:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 4 previous similar messages [ 1269.904542] LustreError: 25100:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743497125 with bad export cookie 5624132132395735078 [ 1269.916834] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1269.919397] LustreError: 25100:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1270.600798] Lustre: server umount lustre-MDT0001 complete [ 1286.889413] Lustre: server umount lustre-OST0000 complete [ 1288.032123] Lustre: 3676:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743497127/real 1743497127] req@ffff9cded6429d00 x1828188136454400/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1743497143 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 1288.080694] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1288.103125] Lustre: Skipped 2 previous similar messages [ 1291.236893] Lustre: 3678:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743497130/real 1743497130] req@ffff9cded642dc40 x1828188136454656/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1743497146 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 1293.187497] Lustre: server umount lustre-OST0001 complete [ 1328.422343] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 1349.604765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1350.326511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1350.329462] Lustre: Skipped 3 previous similar messages [ 1356.530232] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1371.061677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1371.315446] Lustre: 28754:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1371.320747] Lustre: 28754:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 2 previous similar messages [ 1377.684849] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1382.677420] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 1382.680120] Lustre: Skipped 1 previous similar message [ 1393.904437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1395.628810] LustreError: 29507:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1395.661825] LustreError: 29507:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 8 previous similar messages [ 1404.321950] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1418.005721] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1428.709955] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1442.189809] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1463.789459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1463.801663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1463.816142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1469.074987] Lustre: server umount lustre-MDT0000 complete [ 1475.327388] LustreError: 28028:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743497330 with bad export cookie 5624132132395736373 [ 1475.346122] LustreError: 28028:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1475.372570] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1476.010681] Lustre: server umount lustre-MDT0001 complete [ 1493.399034] Lustre: server umount lustre-OST0000 complete [ 1495.660105] Lustre: 3676:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743497335/real 1743497335] req@ffff9cdfd2379d00 x1828188136526080/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1743497351 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 1499.540850] Lustre: server umount lustre-OST0001 complete [ 1516.600923] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 04:49:30 (1743497370) [ 1566.727784] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 1586.776140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1594.501337] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1608.520342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1608.772750] Lustre: 34507:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1608.778126] Lustre: 34507:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 2 previous similar messages [ 1609.280941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1609.292476] Lustre: Skipped 4 previous similar messages [ 1615.529909] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1631.570821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1641.175924] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1654.155741] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1661.961494] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1674.691592] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1680.607562] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 1680.610065] Lustre: Skipped 2 previous similar messages [ 1693.675681] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1693.679121] Lustre: Skipped 4 previous similar messages [ 1693.681233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1693.684015] Lustre: Skipped 4 previous similar messages [ 1699.098151] Lustre: server umount lustre-MDT0000 complete [ 1703.909142] LustreError: 33796:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1703.941650] LustreError: 33796:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 35 previous similar messages [ 1704.373947] LustreError: 33783:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743497559 with bad export cookie 5624132132395737479 [ 1704.390810] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1704.401442] LustreError: 33783:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1705.025059] Lustre: server umount lustre-MDT0001 complete [ 1720.777537] Lustre: server umount lustre-OST0000 complete [ 1725.920163] Lustre: 3676:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743497564/real 1743497564] req@ffff9cded47b4540 x1828188136592256/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1743497580 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 1726.998245] Lustre: server umount lustre-OST0001 complete [ 1743.884396] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 04:53:17 (1743497597) [ 1774.321956] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 1792.363669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1798.987160] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1812.141253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1818.695483] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1832.413237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1840.775581] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1853.305251] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1862.443622] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1873.864708] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1894.885910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1894.895346] Lustre: Skipped 5 previous similar messages [ 1894.903820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1896.924199] Lustre: server umount lustre-MDT0000 complete [ 1901.969589] LustreError: 38879:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743497757 with bad export cookie 5624132132395738585 [ 1901.974463] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1901.979192] LustreError: 38879:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1916.384148] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1916.814955] Lustre: server umount lustre-MDT0001 complete [ 1928.554192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1947.108134] LustreError: 3675:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9cdfe6fbe7c0 x1828188136661504/t0(0) o250->MGC192.168.206.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 1953.232082] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1965.183989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1965.286673] Lustre: 43449:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1965.292223] Lustre: 43449:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 5 previous similar messages [ 1967.685075] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.206.107@tcp (at 0@lo) [ 1970.452389] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1979.069251] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1982.513226] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1988.890814] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1992.032705] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2001.385184] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2001.394620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2001.397572] Lustre: Skipped 24 previous similar messages [ 2005.181232] Lustre: server umount lustre-MDT0000 complete [ 2010.113471] LustreError: 38880:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743497865 with bad export cookie 5624132132395739775 [ 2010.118483] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2010.118526] LustreError: 38880:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2010.491608] Lustre: server umount lustre-MDT0001 complete [ 2026.037304] Lustre: server umount lustre-OST0000 complete [ 2030.056209] Lustre: 3676:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743497869/real 1743497869] req@ffff9cdfe6fe8600 x1828188136701440/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1743497885 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 2031.436447] Lustre: server umount lustre-OST0001 complete [ 2048.439734] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 04:58:21 (1743497901) [ 2078.419460] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 2094.063306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2100.261431] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2112.918967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2119.322663] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2132.530757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2133.045170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2133.053046] Lustre: Skipped 10 previous similar messages [ 2140.270309] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2150.797697] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2158.902629] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2169.495797] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2200.546633] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2200.553026] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2200.562973] Lustre: Skipped 9 previous similar messages [ 2204.518204] Lustre: server umount lustre-MDT0000 complete [ 2209.381831] LustreError: 47338:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743498064 with bad export cookie 5624132132395740552 [ 2209.392271] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2209.889808] Lustre: server umount lustre-MDT0001 complete [ 2225.150044] Lustre: server umount lustre-OST0000 complete [ 2239.612183] Lustre: server umount lustre-OST0001 complete [ 2253.331847] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 05:01:47 (1743498107) [ 2257.159197] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 2261.058887] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 05:01:54 (1743498114) [ 2273.995565] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 2289.679571] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 2305.710679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2314.177156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2322.720826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2322.867376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2323.315898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2323.323143] Lustre: Skipped 6 previous similar messages [ 2323.387561] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2323.513703] Lustre: lustre-MDT0000: new disk, initializing [ 2323.714214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2328.681526] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2337.949790] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2346.732677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2346.847171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2347.152164] Lustre: lustre-OST0000: new disk, initializing [ 2347.156435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2347.159049] Lustre: Skipped 1 previous similar message [ 2348.828189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2348.838186] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2348.893205] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2353.846237] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2363.640659] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2367.972660] Lustre: DEBUG MARKER: create test files [ 2405.348888] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2411.672134] Lustre: server umount lustre-OST0000 complete [ 2416.597356] Lustre: server umount lustre-MDT0000 complete [ 2439.435327] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 2444.835215] Key type lgssc unregistered [ 2445.346548] LNet: 54703:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2446.388636] LNet: Removed LNI 192.168.206.107@tcp [ 2448.041099] Key type .llcrypt unregistered [ 2448.042704] Key type ._llcrypt unregistered [ 2488.801339] Key type ._llcrypt registered [ 2488.802859] Key type .llcrypt registered [ 2488.965711] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 2491.387460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2491.955646] alg: No test for adler32 (adler32-zlib) [ 2493.333731] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 2493.758184] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 2493.764683] LNet: Accept secure, port 988 [ 2495.488193] Key type lgssc registered [ 2496.919580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2506.933672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2507.003473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2508.594408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2513.162989] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2520.647463] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2528.995480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2529.245539] Lustre: 57285:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2529.503204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2533.622340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 2533.750925] Lustre: 57293:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 2533.755152] Lustre: 57293:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 2533.760431] Lustre: 57293:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 2533.764717] Lustre: 57293:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 2533.768322] Lustre: 57293:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 2533.772542] Lustre: 57293:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2536.187177] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2544.094692] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2551.771436] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 2559.740811] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 2569.704970] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2569.709637] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2569.886296] Lustre: server umount lustre-OST0000 complete [ 2574.857218] Lustre: server umount lustre-MDT0000 complete [ 2595.512995] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 2600.371266] Key type lgssc unregistered [ 2600.933815] LNet: 58782:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2601.960645] LNet: Removed LNI 192.168.206.107@tcp [ 2603.269481] Key type .llcrypt unregistered [ 2603.271250] Key type ._llcrypt unregistered [ 2644.915182] Key type ._llcrypt registered [ 2644.916517] Key type .llcrypt registered [ 2645.075791] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 2646.757430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2647.236612] alg: No test for adler32 (adler32-zlib) [ 2648.617235] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 2649.082048] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 2649.092439] LNet: Accept secure, port 988 [ 2650.928216] Key type lgssc registered [ 2652.357926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2662.504529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2662.572463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2664.101127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2668.975755] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2677.079849] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2685.197474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2685.415625] Lustre: 61462:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2685.594718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2690.672755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 2690.760324] Lustre: 61469:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 2690.765158] Lustre: 61469:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 2690.769655] Lustre: 61469:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 2690.773892] Lustre: 61469:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 2690.777434] Lustre: 61469:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 2690.781890] Lustre: 61469:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2692.662935] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2699.995437] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2707.323334] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 2721.762684] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2721.765762] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2721.779272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2724.790991] Lustre: server umount lustre-OST0000 complete [ 2729.982592] Lustre: server umount lustre-MDT0000 complete [ 2753.285639] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 2757.999577] Key type lgssc unregistered [ 2758.433619] LNet: 62916:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2759.464795] LNet: Removed LNI 192.168.206.107@tcp [ 2760.723709] Key type .llcrypt unregistered [ 2760.726759] Key type ._llcrypt unregistered [ 2783.021171] Key type ._llcrypt registered [ 2783.026239] Key type .llcrypt registered [ 2783.154787] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 2806.336322] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 2808.746252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2808.781286] alg: No test for adler32 (adler32-zlib) [ 2810.209962] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 2810.607980] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 2810.614303] LNet: Accept secure, port 988 [ 2812.352217] Key type lgssc registered [ 2813.775157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2826.625364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2836.321402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2845.676270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2854.973327] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2863.042174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2863.112739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2863.133395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2864.456701] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2864.510989] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2864.566570] Lustre: lustre-MDT0000: new disk, initializing [ 2864.704621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2864.745751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2869.186584] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2880.701855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2880.792267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2880.969646] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2881.009184] Lustre: 66796:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2881.056985] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2881.062517] Lustre: Skipped 1 previous similar message [ 2881.158329] Lustre: lustre-MDT0001: new disk, initializing [ 2881.304792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2881.353500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2881.363441] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2885.512851] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2894.700621] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2899.888450] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2907.346670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2907.448172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2907.741315] Lustre: 67746:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2907.773478] Lustre: lustre-OST0000: new disk, initializing [ 2907.781267] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2907.887466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2913.882541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2913.894442] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2914.058832] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2914.430883] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2923.323613] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2927.688960] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2927.969650] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2932.416529] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2932.728255] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2939.888056] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2939.891299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2939.906664] Lustre: Skipped 1 previous similar message [ 2939.916199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2944.997786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2945.003675] Lustre: Skipped 2 previous similar messages [ 2949.088265] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2949.248152] Lustre: server umount lustre-OST0000 complete [ 2955.234692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2955.252250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2955.256663] Lustre: Skipped 1 previous similar message [ 2959.592167] Lustre: server umount lustre-MDT0000 complete [ 2963.639413] LustreError: 66026:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743498818 with bad export cookie 2906728413832292970 [ 2963.643645] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2963.648896] LustreError: 66026:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2964.002417] Lustre: server umount lustre-MDT0001 complete [ 2976.762561] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 05:13:51 (1743498831) [ 2993.548313] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 3012.135646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3042.854413] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 3058.019690] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 3071.022119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3079.702275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3086.869196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3095.270452] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3102.271279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3102.347103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3102.602225] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3102.651596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3102.712457] Lustre: lustre-MDT0000: new disk, initializing [ 3102.835587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3102.878495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3107.236520] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3118.375319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3118.483569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3118.633172] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3118.657731] Lustre: 74374:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3118.716158] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3118.719281] Lustre: Skipped 1 previous similar message [ 3118.778239] Lustre: lustre-MDT0001: new disk, initializing [ 3118.899913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3118.946561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3118.962805] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3123.762641] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3132.716290] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3137.542494] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3145.241700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3145.333705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3145.618266] Lustre: 75326:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3145.653728] Lustre: lustre-OST0000: new disk, initializing [ 3145.662768] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3145.761595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3151.251899] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3153.459817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3153.468925] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3153.566264] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3159.962356] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3164.471904] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3164.769766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3169.516182] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3169.830181] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3173.858754] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3173.862235] LustreError: Skipped 1 previous similar message [ 3173.864401] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3173.871447] Lustre: Skipped 1 previous similar message [ 3173.881483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3178.312334] Lustre: server umount lustre-OST0000 complete [ 3184.612153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3184.618041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3184.622686] Lustre: Skipped 2 previous similar messages [ 3184.625249] Lustre: Skipped 2 previous similar messages [ 3194.849687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3194.854968] Lustre: Skipped 4 previous similar messages [ 3197.408098] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3197.621525] Lustre: server umount lustre-MDT0000 complete [ 3198.951552] LustreError: 73624:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3198.968447] LustreError: 73624:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 4 previous similar messages [ 3201.915412] LustreError: 73606:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743499057 with bad export cookie 2906728413832293852 [ 3201.922778] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3201.924502] LustreError: 73606:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3202.354358] Lustre: server umount lustre-MDT0001 complete [ 3215.190956] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 05:17:49 (1743499069) [ 3222.997686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3223.467513] LustreError: 77600:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3223.554982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3228.090753] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3228.641739] LustreError: 77600:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3232.742166] LustreError: 77600:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3237.857965] LustreError: 77600:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3238.341505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3238.473192] Lustre: 78298:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3243.414267] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3250.307632] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3255.765152] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3263.400394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3263.586738] Lustre: 79181:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3263.783135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3263.793400] Lustre: Skipped 1 previous similar message [ 3269.754558] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3276.999815] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3281.651695] Lustre: 79921:0:(genops.c:1678:obd_export_evict_by_uuid()) MGS: evicting a36f34e4-08fe-446b-977c-f36afae9e3eb at adminstrative request [ 3283.426919] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3283.447260] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 3293.684306] Lustre: MGS: Client a36f34e4-08fe-446b-977c-f36afae9e3eb (at 0@lo) reconnecting [ 3293.690944] Lustre: Evicted from MGS (at 192.168.206.107@tcp) after server handle changed from 0x2856c61785023147 to 0x2856c617850233ee [ 3293.710197] Lustre: MGC192.168.206.107@tcp: Connection restored to 192.168.206.107@tcp (at 0@lo) [ 3320.065699] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3320.181212] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3320.360142] Lustre: 80317:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3320.403886] Lustre: lustre-OST0001: new disk, initializing [ 3320.418162] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3326.424333] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3330.127098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3330.135399] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3330.248158] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3334.573334] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3340.782246] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3340.783891] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3340.800326] LustreError: Skipped 1 previous similar message [ 3340.803825] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3340.810024] Lustre: Skipped 1 previous similar message [ 3343.884701] Lustre: server umount lustre-OST0001 complete [ 3345.901767] LustreError: 79188:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3345.909454] LustreError: 79188:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 3349.987578] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3349.993535] LustreError: Skipped 1 previous similar message [ 3349.996553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3350.007436] Lustre: Skipped 1 previous similar message [ 3362.272867] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3362.479280] Lustre: server umount lustre-OST0000 complete [ 3366.882482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3366.885927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3366.891351] Lustre: Skipped 1 previous similar message [ 3372.962141] Lustre: server umount lustre-MDT0000 complete [ 3376.611602] LustreError: 77600:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3376.624766] LustreError: 77600:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 7 previous similar messages [ 3376.853869] LustreError: 77580:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743499232 with bad export cookie 2906728413832295406 [ 3376.860748] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3377.221551] Lustre: server umount lustre-MDT0001 complete [ 3389.020183] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 05:20:43 (1743499243) [ 3391.508342] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 3394.244685] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 05:20:48 (1743499248) [ 3396.554089] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 3399.112851] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 05:20:53 (1743499253) [ 3401.190936] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 3403.509454] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 05:20:58 (1743499258) [ 3405.876826] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 3411.639789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3412.595383] loop: module loaded [ 3412.610624] loop0: detected capacity change from 0 to 4294967296 [ 3412.667356] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3414.847683] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 3417.773967] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 3422.396703] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 3430.741188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3434.404428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3438.196347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3442.158651] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3449.114401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3449.232405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3449.264491] LustreError: 84241:0:(obd_config.c:780:class_setup()) setup lustre-OST0000-osd failed (-28) [ 3449.267878] LustreError: 84241:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 3449.270711] LustreError: 84241:0:(tgt_mount.c:2234:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 3449.274402] LustreError: 84241:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 3468.941481] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 3483.263971] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 3496.031641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3504.192874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3511.396673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3518.704839] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3525.282661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3525.357367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3525.616605] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3525.659959] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3525.732109] Lustre: lustre-MDT0000: new disk, initializing [ 3525.827905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3525.833416] Lustre: Skipped 1 previous similar message [ 3525.865088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3529.686896] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3539.660133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3539.754182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3539.886809] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3539.908895] Lustre: 87854:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3540.135243] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3543.862357] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3551.466792] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3555.747366] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3562.354790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3562.436059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3567.652616] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3569.189892] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3569.343155] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3575.314944] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3579.552565] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3579.782069] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3583.663251] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3583.907612] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3589.603439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3589.607733] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3589.615970] Lustre: Skipped 1 previous similar message [ 3589.619819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3589.622732] Lustre: Skipped 10 previous similar messages [ 3599.840156] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3600.001771] Lustre: server umount lustre-OST0000 complete [ 3609.702808] Lustre: server umount lustre-MDT0000 complete [ 3610.609154] LustreError: 87100:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3610.623207] LustreError: 87100:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 3612.989525] LustreError: 88829:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743499468 with bad export cookie 2906728413832296001 [ 3612.993836] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3612.999870] LustreError: 88829:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3613.265250] Lustre: server umount lustre-MDT0001 complete [ 3623.839678] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 05:24:38 (1743499478) [ 3630.029909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3634.074179] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3641.723448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3641.815265] Lustre: 91775:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3641.820524] Lustre: 91775:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [ 3645.370025] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3651.289356] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3655.549167] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3661.983176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3662.328284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3662.333692] Lustre: Skipped 4 previous similar messages [ 3667.997767] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3673.755500] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3677.418978] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3677.662178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3681.277754] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3681.526624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3687.986332] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3688.071760] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3688.251522] Lustre: lustre-OST0001: new disk, initializing [ 3688.253417] Lustre: Skipped 2 previous similar messages [ 3688.257200] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3688.259733] Lustre: Skipped 3 previous similar messages [ 3693.855455] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3697.223947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3697.231042] Lustre: Skipped 2 previous similar messages [ 3697.241031] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3697.365513] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3701.434299] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3705.202292] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3705.431608] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3709.051900] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3709.281200] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3726.709325] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3754.151145] Lustre: Failing over lustre-MDT0000 [ 3754.571112] Lustre: server umount lustre-MDT0000 complete [ 3754.978909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3754.982471] LustreError: Skipped 1 previous similar message [ 3754.984637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3754.991954] Lustre: Skipped 3 previous similar messages [ 3754.995187] LustreError: 91078:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3755.007216] LustreError: 91078:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 3775.969236] Lustre: 64251:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743499614/real 1743499614] req@000000004a6321aa x1828190856422016/t0(0) o400->MGC192.168.206.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1743499630 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 3775.984172] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3780.348127] LDISKFS-fs (dm-0): recovery complete [ 3780.354537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3786.208636] LustreError: 64247:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bb041f05 x1828190856430336/t0(0) o250->MGC192.168.206.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 3786.555630] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3789.307152] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3789.314081] Lustre: 93911:0:(ldlm_lib.c:1971:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 3790.613202] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3791.853772] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.206.107@tcp (at 0@lo) [ 3791.860885] Lustre: 91078:0:(ldlm_lib.c:1971:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 3791.872281] Lustre: 91078:0:(ldlm_lib.c:1971:extend_recovery_timer()) Skipped 1 previous similar message [ 3791.951223] LustreError: 96687:0:(ldlm_lib.c:2604:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3792.273195] LustreError: 96687:0:(ldlm_lib.c:2604:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3792.600120] LustreError: 96687:0:(ldlm_lib.c:2604:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3792.606642] LustreError: 96687:0:(ldlm_lib.c:2604:replay_request_or_update()) Skipped 1 previous similar message [ 3792.936143] LustreError: 96687:0:(ldlm_lib.c:2604:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3792.940757] LustreError: 96687:0:(ldlm_lib.c:2604:replay_request_or_update()) Skipped 1 previous similar message [ 3792.952172] Lustre: 96687:0:(ldlm_lib.c:1971:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3792.962661] Lustre: 96687:0:(ldlm_lib.c:1971:extend_recovery_timer()) Skipped 7 previous similar messages [ 3793.601916] LustreError: 96687:0:(ldlm_lib.c:2604:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3793.606467] LustreError: 96687:0:(ldlm_lib.c:2604:replay_request_or_update()) Skipped 2 previous similar messages [ 3794.248234] LustreError: 96687:0:(ldlm_lib.c:2604:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3794.253883] LustreError: 96687:0:(ldlm_lib.c:2604:replay_request_or_update()) Skipped 3 previous similar messages [ 3795.211737] Lustre: 96687:0:(ldlm_lib.c:1971:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3795.217359] Lustre: 96687:0:(ldlm_lib.c:1971:extend_recovery_timer()) Skipped 6 previous similar messages [ 3795.858488] LustreError: 96687:0:(ldlm_lib.c:2604:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3795.865205] LustreError: 96687:0:(ldlm_lib.c:2604:replay_request_or_update()) Skipped 6 previous similar messages [ 3796.504419] LustreError: 96687:0:(ldlm_lib.c:2604:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3796.512290] LustreError: 96687:0:(ldlm_lib.c:2604:replay_request_or_update()) Skipped 6 previous similar messages [ 3799.361374] Lustre: 96687:0:(ldlm_lib.c:1971:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3799.371714] Lustre: 96687:0:(ldlm_lib.c:1971:extend_recovery_timer()) Skipped 12 previous similar messages [ 3800.020720] LustreError: 96687:0:(ldlm_lib.c:2604:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3800.028980] LustreError: 96687:0:(ldlm_lib.c:2604:replay_request_or_update()) Skipped 12 previous similar messages [ 3800.664147] LustreError: 96687:0:(ldlm_lib.c:2604:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3800.668411] LustreError: 96687:0:(ldlm_lib.c:2604:replay_request_or_update()) Skipped 12 previous similar messages [ 3807.674349] Lustre: 96687:0:(ldlm_lib.c:1971:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3807.680743] Lustre: 96687:0:(ldlm_lib.c:1971:extend_recovery_timer()) Skipped 25 previous similar messages [ 3808.322208] LustreError: 96687:0:(ldlm_lib.c:2604:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3808.327824] LustreError: 96687:0:(ldlm_lib.c:2604:replay_request_or_update()) Skipped 25 previous similar messages [ 3808.968765] LustreError: 96687:0:(ldlm_lib.c:2604:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3808.971523] LustreError: 96687:0:(ldlm_lib.c:2604:replay_request_or_update()) Skipped 25 previous similar messages [ 3823.848390] Lustre: 96687:0:(ldlm_lib.c:1971:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3823.866345] Lustre: 96687:0:(ldlm_lib.c:1971:extend_recovery_timer()) Skipped 50 previous similar messages [ 3824.512262] LustreError: 96687:0:(ldlm_lib.c:2604:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3824.517393] LustreError: 96687:0:(ldlm_lib.c:2604:replay_request_or_update()) Skipped 50 previous similar messages [ 3825.160187] LustreError: 96687:0:(ldlm_lib.c:2604:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3825.163871] LustreError: 96687:0:(ldlm_lib.c:2604:replay_request_or_update()) Skipped 50 previous similar messages [ 3849.224184] Lustre: 96687:0:(ldlm_lib.c:2279:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 3849.228702] Lustre: 96687:0:(ldlm_lib.c:2289:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3849.231884] Lustre: 96687:0:(genops.c:1508:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 182cd10d-bcc3-4bc7-8a30-28efbf3fa2f0@192.168.206.7@tcp [ 3849.237327] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3849.244699] Lustre: 96687:0:(ldlm_lib.c:1801:abort_req_replay_queue()) @@@ aborted: req@000000008a7b81da x1828190835249408/t0(8589934777) o101->182cd10d-bcc3-4bc7-8a30-28efbf3fa2f0@192.168.206.7@tcp:110/0 lens 592/0 e 0 to 0 dl 1743499715 ref 1 fl Complete:/204/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 3849.262610] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3849.270486] Lustre: lustre-MDT0000: Denying connection for new client 182cd10d-bcc3-4bc7-8a30-28efbf3fa2f0 (at 192.168.206.7@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) already passed deadline 0:00 [ 3849.286573] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3849.319270] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 192.168.206.107@tcp (at 0@lo) [ 3849.326037] Lustre: Skipped 2 previous similar messages [ 3849.337441] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 3849.362682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:79 to 0x280000401:577) [ 3849.363527] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:47 to 0x2c0000401:545) [ 3865.571769] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3865.580313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3865.582866] Lustre: Skipped 6 previous similar messages [ 3870.305772] Lustre: server umount lustre-OST0000 complete [ 3880.147745] Lustre: server umount lustre-OST0001 complete [ 3884.005933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3884.014915] Lustre: Skipped 8 previous similar messages [ 3898.336149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3898.541360] Lustre: server umount lustre-MDT0000 complete [ 3899.361705] LustreError: 91079:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3899.372123] LustreError: 91079:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 47 previous similar messages [ 3902.062246] LustreError: 91058:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743499757 with bad export cookie 2906728413832347220 [ 3902.068496] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3902.069111] LustreError: 91058:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3902.396780] Lustre: server umount lustre-MDT0001 complete [ 3913.878380] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 05:29:28 (1743499768) [ 3921.365818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3921.370401] Lustre: *** cfs_fail_loc=197, val=0*** [ 3938.016823] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3944.160756] LustreError: 98642:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.107@tcp: failed processing log, type 4: rc = -110 [ 3950.347496] Lustre: *** cfs_fail_loc=197, val=0*** [ 3950.352065] Lustre: Skipped 65 previous similar messages [ 3973.987916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3973.990857] Lustre: Skipped 2 previous similar messages [ 3979.762571] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3986.162288] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3988.322654] LustreError: 64247:0:(client.c:1282:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000000da3a9eb x1828190856484864/t0(0) o38->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 3988.516974] Lustre: server umount lustre-OST0000 complete [ 4000.178121] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 05:30:54 (1743499854) [ 4021.350478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4025.892916] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4034.575863] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4039.033591] LustreError: 100176:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4039.040442] LustreError: 100176:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4039.048016] LustreError: 100176:0:(lod_dev.c:506:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 4039.395626] Lustre: server umount lustre-MDT0000 complete [ 4056.640300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4073.558621] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 4088.178175] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 4102.384072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4111.002953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4118.281853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4125.259903] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4131.666909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4131.743326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4132.042637] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4132.088901] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4132.149218] Lustre: lustre-MDT0000: new disk, initializing [ 4132.263629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4136.091901] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4146.026396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4146.159183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4146.322131] Lustre: 104982:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4146.330355] Lustre: 104982:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 2 previous similar messages [ 4146.568125] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4150.309228] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4158.217254] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4162.818969] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4170.485893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4170.618972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4176.867383] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4179.621932] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4184.810681] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4188.949745] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4189.237560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4192.938736] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4193.214577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4195.298900] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4195.300679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4195.301855] LustreError: Skipped 3 previous similar messages [ 4195.303194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4195.303198] Lustre: Skipped 17 previous similar messages [ 4195.319299] Lustre: Skipped 1 previous similar message [ 4201.132449] Lustre: server umount lustre-OST0000 complete [ 4219.360153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4219.529630] Lustre: server umount lustre-MDT0000 complete [ 4220.901043] LustreError: 105129:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4220.907238] LustreError: 105129:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 5 previous similar messages [ 4223.608857] LustreError: 104211:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743500078 with bad export cookie 2906728413832357853 [ 4223.610423] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4223.620448] LustreError: 104211:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4223.984629] Lustre: server umount lustre-MDT0001 complete [ 4235.471186] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 05:34:50 (1743500090) [ 4252.040768] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 4255.987913] Key type lgssc unregistered [ 4256.369817] LNet: 108277:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4257.384521] LNet: Removed LNI 192.168.206.107@tcp [ 4258.316498] Key type .llcrypt unregistered [ 4258.318831] Key type ._llcrypt unregistered [ 4276.543044] Key type ._llcrypt registered [ 4276.545304] Key type .llcrypt registered [ 4276.662461] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 4295.633718] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 4298.090693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4298.162913] alg: No test for adler32 (adler32-zlib) [ 4299.482088] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 4299.875300] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 4299.878800] LNet: Accept secure, port 988 [ 4301.584211] Key type lgssc registered [ 4302.845372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4314.074729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4321.847668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4328.487166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4335.992443] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4342.604394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4349.533727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4349.640352] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4349.658182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4351.014152] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4351.070544] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4351.133538] Lustre: lustre-MDT0000: new disk, initializing [ 4351.227211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4351.261500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4354.932834] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4365.061731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4365.129591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4365.368144] Lustre: 112371:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4365.407818] Lustre: lustre-OST0000: new disk, initializing [ 4365.413245] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4365.416070] Lustre: Skipped 1 previous similar message [ 4365.508534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4370.657977] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4374.598169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 4374.602614] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 4374.641524] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 4380.951594] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4381.057781] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4381.220751] Lustre: 113225:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4381.256421] Lustre: lustre-OST0001: new disk, initializing [ 4381.262682] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4381.331680] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4387.054986] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4387.999903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 4388.014110] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 4388.047704] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 4398.056157] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4398.057493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4398.061099] Lustre: Skipped 1 previous similar message [ 4398.066750] Lustre: Skipped 1 previous similar message [ 4402.469991] Lustre: server umount lustre-MDT0000 complete [ 4408.830375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4413.189607] LustreError: 112394:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743500268 with bad export cookie 13944041866931688477 [ 4413.190763] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4413.193280] LustreError: 112394:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4413.262169] Lustre: server umount lustre-OST0000 complete [ 4416.791726] Lustre: server umount lustre-OST0001 complete [ 4427.628907] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 05:38:02 (1743500282) [ 4441.021961] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 4452.119924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4453.153293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4459.721329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4460.892086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4470.294728] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 05:38:44 (1743500324) [ 4483.339544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4485.250900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4487.262833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4489.207195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4490.095905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4491.987059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4508.181763] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 4521.781511] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 4534.485214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4541.104969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4547.439364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4554.514037] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4561.193244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4561.277204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4561.586936] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4561.644388] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4561.708498] Lustre: lustre-MDT0000: new disk, initializing [ 4561.813671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4561.839753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4565.203615] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4574.329094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4574.413048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4574.529985] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4574.551901] Lustre: 120729:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4574.585057] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4574.587630] Lustre: Skipped 1 previous similar message [ 4574.644337] Lustre: lustre-MDT0001: new disk, initializing [ 4574.751180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4574.789438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4574.796785] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4578.089183] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4585.592812] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4589.707949] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4595.959685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4596.064117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4596.301529] Lustre: 121678:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4596.338519] Lustre: lustre-OST0000: new disk, initializing [ 4596.342486] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4596.439671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4601.580561] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4605.007271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4605.019247] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4605.177923] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4609.440420] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4613.532521] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4613.760715] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4617.419536] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4617.679199] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4620.772911] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4620.776655] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4620.779110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4620.788871] Lustre: Skipped 1 previous similar message [ 4620.793785] Lustre: Skipped 1 previous similar message [ 4625.514755] Lustre: server umount lustre-OST0000 complete [ 4631.011693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4631.014378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4631.019776] LustreError: Skipped 1 previous similar message [ 4631.029318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4631.032272] Lustre: Skipped 1 previous similar message [ 4635.322231] Lustre: server umount lustre-MDT0000 complete [ 4639.088315] LustreError: 120743:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743500494 with bad export cookie 13944041866931689702 [ 4639.092617] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4639.093712] LustreError: 120743:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4639.380824] Lustre: server umount lustre-MDT0001 complete [ 4649.528626] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 05:41:44 (1743500504) [ 4656.219267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4656.698832] LustreError: 123952:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4656.716025] LustreError: 123952:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 4656.779522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4660.574481] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4662.247965] LustreError: 123952:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4667.363167] LustreError: 123952:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4668.063157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4668.146274] Lustre: 124648:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4671.340849] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4676.416963] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4680.387286] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4686.289399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4686.459787] Lustre: 125529:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4691.357970] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4696.778822] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4706.805699] Lustre: *** cfs_fail_loc=159, val=0*** [ 4723.220996] Lustre: lustre-MDT0000: Client 6ef5a9a9-505d-4ec0-9653-47a351e00aae (at 192.168.206.7@tcp) reconnecting [ 4724.747192] Lustre: *** cfs_fail_loc=159, val=0*** [ 4724.748939] Lustre: Skipped 5 previous similar messages [ 4741.160549] Lustre: lustre-MDT0000: Client 6ef5a9a9-505d-4ec0-9653-47a351e00aae (at 192.168.206.7@tcp) reconnecting [ 4747.647133] Lustre: server umount lustre-OST0000 complete [ 4753.380855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4753.381794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4753.388711] Lustre: Skipped 2 previous similar messages [ 4758.499282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4758.502307] Lustre: Skipped 2 previous similar messages [ 4761.681780] Lustre: server umount lustre-MDT0000 complete [ 4763.617982] LustreError: 126093:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4763.625132] LustreError: 126093:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 4764.640168] Lustre: 109607:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743500603/real 1743500603] req@0000000009e378ce x1828192418632192/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1743500619 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 4764.653700] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4765.134074] LustreError: 123932:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743500620 with bad export cookie 13944041866931690584 [ 4765.140043] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4765.143625] LustreError: 123932:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4765.480350] Lustre: server umount lustre-MDT0001 complete [ 4782.875378] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 4786.437889] Key type lgssc unregistered [ 4786.785989] LNet: 127372:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4787.814888] LNet: Removed LNI 192.168.206.107@tcp [ 4788.787779] Key type .llcrypt unregistered [ 4788.789331] Key type ._llcrypt unregistered [ 4818.331714] Key type ._llcrypt registered [ 4818.334211] Key type .llcrypt registered [ 4818.442322] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 4820.728884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4820.761622] alg: No test for adler32 (adler32-zlib) [ 4822.099747] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 4822.470906] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 4822.473853] LNet: Accept secure, port 988 [ 4824.160225] Key type lgssc registered [ 4825.286605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4832.744322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4832.767088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4834.187094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4837.428881] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4844.515055] LustreError: 128758:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4844.529852] LustreError: 128758:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 4844.583835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4844.666114] Lustre: 129455:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4844.918838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4848.388650] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4854.081430] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4858.449399] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4865.028107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4865.227976] Lustre: 130337:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4865.396255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4870.199860] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 4875.366731] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4878.702157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4888.687170] Lustre: *** cfs_fail_loc=159, val=0*** [ 4904.971869] Lustre: lustre-MDT0000: Client 2abd6e60-d121-4bf9-a531-a6b952f96232 (at 192.168.206.7@tcp) reconnecting [ 4907.295918] Lustre: *** cfs_fail_loc=159, val=0*** [ 4922.906856] Lustre: lustre-MDT0001: Client 2abd6e60-d121-4bf9-a531-a6b952f96232 (at 192.168.206.7@tcp) reconnecting [ 4924.476337] Lustre: *** cfs_fail_loc=159, val=0*** [ 4924.477849] Lustre: Skipped 3 previous similar messages [ 4940.815396] Lustre: lustre-MDT0001: Client 2abd6e60-d121-4bf9-a531-a6b952f96232 (at 192.168.206.7@tcp) reconnecting [ 4944.694851] Lustre: 131443:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4947.451738] Lustre: *** cfs_fail_loc=159, val=0*** [ 4947.454940] Lustre: Skipped 5 previous similar messages [ 4963.850556] Lustre: lustre-MDT0000: Client 4e55f0d6-e52e-4a54-b44b-539866fccaeb (at 192.168.206.7@tcp) reconnecting [ 4965.449037] Lustre: *** cfs_fail_loc=159, val=0*** [ 4965.451484] Lustre: Skipped 12 previous similar messages [ 4980.756237] Lustre: lustre-MDT0000: Client 4e55f0d6-e52e-4a54-b44b-539866fccaeb (at 192.168.206.7@tcp) reconnecting [ 4982.350408] Lustre: 131736:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4987.878592] LustreError: lustre-OST0000-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 4987.881670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4987.888660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4991.460475] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4991.460994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4991.463033] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4992.693737] Lustre: server umount lustre-OST0000 complete [ 4996.578106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4996.584018] Lustre: Skipped 1 previous similar message [ 4996.586269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4996.588803] Lustre: Skipped 2 previous similar messages [ 5001.698080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5001.702741] Lustre: Skipped 1 previous similar message [ 5002.252509] Lustre: server umount lustre-MDT0000 complete [ 5005.974783] LustreError: 128737:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743500861 with bad export cookie 300492747470922909 [ 5005.976925] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5005.981615] LustreError: 128737:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5006.255030] Lustre: server umount lustre-MDT0001 complete [ 5022.584943] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 5025.874958] Key type lgssc unregistered [ 5026.241921] LNet: 132783:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5027.304702] LNet: Removed LNI 192.168.206.107@tcp [ 5028.352452] Key type .llcrypt unregistered [ 5028.354096] Key type ._llcrypt unregistered [ 5058.292495] Key type ._llcrypt registered [ 5058.294224] Key type .llcrypt registered [ 5058.413500] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 5060.646962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5060.709157] alg: No test for adler32 (adler32-zlib) [ 5061.957472] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 5062.340596] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 5062.343748] LNet: Accept secure, port 988 [ 5064.056214] Key type lgssc registered [ 5065.374651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5072.929565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5072.954710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5074.447479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5077.851408] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5084.642677] LustreError: 134167:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5084.649381] LustreError: 134167:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 5085.272364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5085.353783] Lustre: 134864:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5085.591284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5088.651715] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5093.723754] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5098.197408] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5104.703141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5104.928577] Lustre: 135748:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5105.095836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5110.284231] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 5110.497295] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5116.864484] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5119.493678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 5127.350476] Lustre: 136398:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 5129.865633] Lustre: 136447:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 5131.600212] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 5132.490328] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight in log params [ 5135.998591] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 5137.780399] Lustre: 136693:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 5145.067177] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5145.068041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5145.077264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5145.081910] Lustre: Skipped 1 previous similar message [ 5145.087931] Lustre: Skipped 1 previous similar message [ 5147.481386] Lustre: server umount lustre-OST0000 complete [ 5152.231929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5152.236558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5152.247289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5155.297434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5155.300202] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5157.400879] Lustre: server umount lustre-MDT0000 complete [ 5160.417945] LustreError: 134823:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5160.428632] LustreError: 134823:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 5160.925538] LustreError: 134148:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743501016 with bad export cookie 5225901062449150358 [ 5160.931173] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5160.933904] LustreError: 134148:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5161.236454] Lustre: server umount lustre-MDT0001 complete [ 5177.667041] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 5181.212523] Key type lgssc unregistered [ 5181.581861] LNet: 137750:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5182.634209] LNet: Removed LNI 192.168.206.107@tcp [ 5183.747483] Key type .llcrypt unregistered [ 5183.749039] Key type ._llcrypt unregistered [ 5213.424694] Key type ._llcrypt registered [ 5213.426386] Key type .llcrypt registered [ 5213.529683] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 5215.330503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5215.349974] alg: No test for adler32 (adler32-zlib) [ 5216.545562] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 5216.771405] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 5216.777872] LNet: Accept secure, port 988 [ 5218.472284] Key type lgssc registered [ 5219.660215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5226.973487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5227.005937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5228.481508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5232.067247] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5237.731337] LustreError: 139139:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5237.738131] LustreError: 139139:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 5239.486074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5239.574550] Lustre: 139838:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5239.844731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5243.336461] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5248.676451] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5253.246399] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5260.070630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5260.321737] Lustre: 140723:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5260.471114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5265.585996] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 5265.844688] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5271.579241] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5272.764360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 5277.689262] Lustre: *** cfs_fail_loc=159, val=0*** [ 5294.100443] Lustre: lustre-MDT0000: Client ec8d6fc3-e219-40a6-b530-e622ef7a0b85 (at 192.168.206.7@tcp) reconnecting [ 5311.969720] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5311.975189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5311.981523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5314.021235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5314.022172] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5314.033577] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5316.828849] Lustre: server umount lustre-OST0000 complete [ 5321.697619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5321.701853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5321.708176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5321.710916] Lustre: Skipped 1 previous similar message [ 5324.258009] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5324.264526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5324.270816] Lustre: Skipped 1 previous similar message [ 5325.710440] Lustre: server umount lustre-MDT0000 complete [ 5328.512919] LustreError: 139118:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743501183 with bad export cookie 13222041126018250687 [ 5328.516895] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5328.518910] LustreError: 139118:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5328.785792] Lustre: server umount lustre-MDT0001 complete [ 5343.947549] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 5347.409606] Key type lgssc unregistered [ 5347.742516] LNet: 142446:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5348.776555] LNet: Removed LNI 192.168.206.107@tcp [ 5349.679307] Key type .llcrypt unregistered [ 5349.680969] Key type ._llcrypt unregistered [ 5377.816830] Key type ._llcrypt registered [ 5377.818424] Key type .llcrypt registered [ 5377.915341] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 5379.450505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5379.468066] alg: No test for adler32 (adler32-zlib) [ 5380.617896] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 5380.926816] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 5380.929685] LNet: Accept secure, port 988 [ 5382.593158] Key type lgssc registered [ 5383.545576] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5389.985819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5390.004668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5391.466648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5394.480048] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5400.924426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5401.014360] Lustre: 144530:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5401.058668] LustreError: 143832:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5401.065408] LustreError: 143832:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 5401.290675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5404.071755] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5409.044696] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5412.807613] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5418.484420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5418.664799] Lustre: 145417:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5418.854323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5423.244376] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5426.035083] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 5428.356499] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5436.273642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 5457.093274] Lustre: lustre-OST0000: evicting cc152746-8d32-4a21-81d0-a0cd5ea1cbd6 (at 192.168.206.7@tcp) by administrative request [ 5457.097895] Lustre: Skipped 1 previous similar message [ 5487.590776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5487.595458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5487.603122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5487.605542] Lustre: Skipped 1 previous similar message [ 5490.459395] Lustre: server umount lustre-OST0000 complete [ 5497.829862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5497.835071] Lustre: Skipped 2 previous similar messages [ 5497.836153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5497.839240] Lustre: Skipped 1 previous similar message [ 5499.493235] Lustre: server umount lustre-MDT0000 complete [ 5502.464616] LustreError: 145447:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743501357 with bad export cookie 7193083581324302567 [ 5502.469177] LustreError: 145447:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5502.470696] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5502.733243] Lustre: server umount lustre-MDT0001 complete [ 5517.904418] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 5521.252459] Key type lgssc unregistered [ 5521.627673] LNet: 147566:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5522.665859] LNet: Removed LNI 192.168.206.107@tcp [ 5523.657898] Key type .llcrypt unregistered [ 5523.659664] Key type ._llcrypt unregistered [ 5618.356552] Key type ._llcrypt registered [ 5618.359575] Key type .llcrypt registered [ 5618.459952] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 5620.165821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5620.181699] alg: No test for adler32 (adler32-zlib) [ 5621.306167] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 5621.534792] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 5621.537483] LNet: Accept secure, port 988 [ 5623.192219] Key type lgssc registered [ 5624.150680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5630.738407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5630.761873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5632.195239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5635.245116] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5641.700327] LustreError: 150028:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5641.707847] LustreError: 150028:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 5641.785988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5641.869902] Lustre: 150728:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5642.084655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5644.868760] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5649.407654] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5652.842425] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5657.731510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5657.861098] Lustre: 151611:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5657.999889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5661.807602] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5666.402595] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5667.251960] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 5677.487035] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 5687.780050] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5687.780960] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5687.782912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5687.784618] LustreError: Skipped 1 previous similar message [ 5687.788975] Lustre: Skipped 1 previous similar message [ 5689.804817] Lustre: server umount lustre-OST0000 complete [ 5692.897906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5692.899274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5692.903416] Lustre: Skipped 1 previous similar message [ 5692.913067] Lustre: Skipped 2 previous similar messages [ 5698.018517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5698.024072] Lustre: Skipped 1 previous similar message [ 5698.314768] Lustre: server umount lustre-MDT0000 complete [ 5700.866505] LustreError: 150010:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743501556 with bad export cookie 13729979615987687214 [ 5700.869288] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5700.871154] LustreError: 150010:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5701.091757] Lustre: server umount lustre-MDT0001 complete [ 5715.733085] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 5718.596736] Key type lgssc unregistered [ 5718.904136] LNet: 153234:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5719.974598] LNet: Removed LNI 192.168.206.107@tcp [ 5720.804924] Key type .llcrypt unregistered [ 5720.806714] Key type ._llcrypt unregistered [ 5745.977166] Key type ._llcrypt registered [ 5745.978556] Key type .llcrypt registered [ 5746.065037] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 5747.570117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5747.590379] alg: No test for adler32 (adler32-zlib) [ 5748.704856] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 5748.914879] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 5748.918171] LNet: Accept secure, port 988 [ 5750.560163] Key type lgssc registered [ 5751.518400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5757.145778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5765.336328] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 06:00:20 (1743501620) [ 5776.198932] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 5786.355299] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 5794.552467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5799.745178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5804.734456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5810.111256] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5814.484334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5814.536225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5814.548190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5815.757336] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5815.787887] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5815.836460] Lustre: lustre-MDT0000: new disk, initializing [ 5815.893521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5815.908397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5818.341088] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5825.275284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5825.347738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5825.448316] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5825.466908] Lustre: 158200:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5825.494056] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5825.495653] Lustre: Skipped 1 previous similar message [ 5825.542871] Lustre: lustre-MDT0001: new disk, initializing [ 5825.628929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5825.658667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5825.662875] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5828.149278] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5833.950449] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5837.438836] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5842.244667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5842.319184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5842.497857] Lustre: 159154:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5842.519063] Lustre: lustre-OST0000: new disk, initializing [ 5842.522287] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5842.588710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5846.226912] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5848.591542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5848.596191] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5848.689104] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5851.847761] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5858.787052] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5858.789672] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5858.791616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5858.793019] Lustre: Skipped 1 previous similar message [ 5863.909553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5863.912545] Lustre: Skipped 2 previous similar messages [ 5864.205623] Lustre: server umount lustre-OST0000 complete [ 5869.027070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5869.029833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5869.034320] Lustre: Skipped 1 previous similar message [ 5872.761072] Lustre: server umount lustre-MDT0000 complete [ 5875.163415] LustreError: 157433:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743501730 with bad export cookie 16313125990072862658 [ 5875.168256] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5875.168650] LustreError: 157433:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5875.402717] Lustre: server umount lustre-MDT0001 complete [ 5888.693131] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 5891.750418] Key type lgssc unregistered [ 5892.034688] LNet: 160957:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5893.094495] LNet: Removed LNI 192.168.206.107@tcp [ 5893.857814] Key type .llcrypt unregistered [ 5893.859488] Key type ._llcrypt unregistered [ 5919.207470] Key type ._llcrypt registered [ 5919.208530] Key type .llcrypt registered [ 5919.290385] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 5921.483465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5921.554086] alg: No test for adler32 (adler32-zlib) [ 5922.865093] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 5923.217710] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 5923.220047] LNet: Accept secure, port 988 [ 5924.976256] Key type lgssc registered [ 5925.878035] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5931.371304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5931.388118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5932.741873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5935.301196] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5940.961379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5941.049258] Lustre: 163038:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5941.244130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5943.759882] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5947.714766] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5950.921582] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5955.489056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5955.608701] Lustre: 163921:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5955.701126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5957.740741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5959.384303] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 5963.466786] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5972.664101] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5972.678832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5972.685398] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5972.689541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.694762] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.206.107@tcp (at 0@lo) [ 5973.383936] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5973.395760] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5973.402378] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5973.404874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.408473] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.206.107@tcp (at 0@lo) [ 5974.364445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.107846] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5975.110927] Lustre: Skipped 1 previous similar message [ 5975.122060] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5975.126194] Lustre: Skipped 1 previous similar message [ 5975.128317] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5975.131036] Lustre: Skipped 1 previous similar message [ 5975.132426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.137724] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.206.107@tcp (at 0@lo) [ 5975.140580] Lustre: Skipped 1 previous similar message [ 5975.710933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.306273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.899429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.494384] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5977.497915] Lustre: Skipped 3 previous similar messages [ 5977.509112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5977.513756] Lustre: Skipped 3 previous similar messages [ 5977.515806] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5977.517429] Lustre: Skipped 3 previous similar messages [ 5977.518884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.522236] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.206.107@tcp (at 0@lo) [ 5977.524207] Lustre: Skipped 3 previous similar messages [ 5978.075112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.700071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.316174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.950098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.504401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.109295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.656295] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5981.658976] Lustre: Skipped 6 previous similar messages [ 5981.671382] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5981.675450] Lustre: Skipped 6 previous similar messages [ 5981.677806] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5981.681766] Lustre: Skipped 6 previous similar messages [ 5981.684160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.688783] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.206.107@tcp (at 0@lo) [ 5981.691536] Lustre: Skipped 6 previous similar messages [ 5982.303100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.886846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.492990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.064790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.678076] LustreError: 162361:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5984.682148] LustreError: 162361:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5984.689928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.305685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.883941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.445924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.059675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.639485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.221283] LustreError: 162361:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5988.225763] LustreError: 162361:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5988.236391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.824451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.416207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.995175] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5989.997718] Lustre: Skipped 13 previous similar messages [ 5990.008381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5990.013482] Lustre: Skipped 13 previous similar messages [ 5990.015435] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5990.018242] Lustre: Skipped 13 previous similar messages [ 5990.019634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.023471] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.206.107@tcp (at 0@lo) [ 5990.026152] Lustre: Skipped 13 previous similar messages [ 5990.660200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.287937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.951331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.548928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.151562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.686327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.263034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.860792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.447185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.058529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.671080] LustreError: 162361:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5996.675047] LustreError: 162361:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5996.681744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.285240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.890671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.461647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.071313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.707254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.334996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.900293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.552837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.128024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.737088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.364076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.995210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.576734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.165061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.756890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.322551] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6006.324744] Lustre: Skipped 26 previous similar messages [ 6006.336373] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6006.340196] Lustre: Skipped 26 previous similar messages [ 6006.341915] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6006.344364] Lustre: Skipped 26 previous similar messages [ 6006.346106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.349590] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.206.107@tcp (at 0@lo) [ 6006.352114] Lustre: Skipped 26 previous similar messages [ 6006.893115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.476972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.037072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.627914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.223666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.796748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.345483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.962465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.510610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.062398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.658462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.215054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.793807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.372125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.039263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.731377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.350934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.949267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.569406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.231206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.847721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.464748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.045605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.690165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.365590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.961173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.554234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.136663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.694901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.268524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.846196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.425273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.001530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.601620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.210886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.806697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.384736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.947916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.470977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.060504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.647334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.295385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.897420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.503855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.118357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.714362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.264352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.801175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.356545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.936099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.524385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.091549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.616575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.192915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.758727] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6038.760460] Lustre: Skipped 54 previous similar messages [ 6038.768765] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6038.771623] Lustre: Skipped 54 previous similar messages [ 6038.773185] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6038.774932] Lustre: Skipped 54 previous similar messages [ 6038.776200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.780408] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.206.107@tcp (at 0@lo) [ 6038.782416] Lustre: Skipped 54 previous similar messages [ 6039.299659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.822403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.389566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.942662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.521039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.068136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.632456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.151276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.677371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.246655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.838339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.367472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.908358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.456889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.060385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.589468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.112441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.659388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.182990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.756980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.338982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.876825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.409113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.965924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.502068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.022912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.572969] LustreError: 162361:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6053.577651] LustreError: 162361:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6053.583628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.111362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.640745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.325811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.857385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.444296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.987075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.542908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.064587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.615400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.151356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.704962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.234962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.878098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.421031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.994590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.540285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.071157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.617518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.188066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.697111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.207448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.745898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.358977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.916389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.533909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.084852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.638762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.201742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.798351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.361059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.951539] LustreError: 162361:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6070.955862] LustreError: 162361:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6070.962317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.507455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.128840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.662757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.246541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.793377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.351367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.893450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.427592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.988461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.541645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.114217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.785277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.333789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.864618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.398876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.939905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.466692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.008465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.569326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.161121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.734307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.335307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.865930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.476633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.060821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.646466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.242819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.854747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.442976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.099545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.651027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.277714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.819449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.343914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.860451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.386143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.940467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.451755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.979558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.550174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.070944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.648534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.190651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.753451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.297632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.838215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.386204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.950841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.499404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.002343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.585666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.172269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.695821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.218331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.728214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.221780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.894117] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6102.896182] Lustre: Skipped 114 previous similar messages [ 6102.906115] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6102.911473] Lustre: Skipped 114 previous similar messages [ 6102.913722] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6102.916942] Lustre: Skipped 114 previous similar messages [ 6102.918444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.921985] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.206.107@tcp (at 0@lo) [ 6102.924580] Lustre: Skipped 114 previous similar messages [ 6103.433336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.929576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.451865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.988580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.530895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.060298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.591030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.097417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.789263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.342790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.910069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.426233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.001995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.517432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.059855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.601528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.181589] LustreError: 162361:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6112.185205] LustreError: 162361:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6112.192380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.716646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.307272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.882631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.447726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.076056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.691054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.275100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.940470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.501769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.101909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.661651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.264813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.858829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.471117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.062092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.694772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.295579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.949983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.540954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.156699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.747959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.359557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.944508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.558791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.200048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.884930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.444900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6129.071985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6129.615551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.195693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.807176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.453546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.016112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.615143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.157188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.750968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.325948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.928709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.449036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.955199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.606414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.179495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.782340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.317418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.923605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.478671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6140.054394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6140.653329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.261847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.807315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.370746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.898138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.450043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.046371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.607473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.151230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.746077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.396196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.038639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.658375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.243464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.794331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.406425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.986921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6150.569185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.129284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.701366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.307491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.872895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6153.431528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6154.018715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6154.573567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.134880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.672534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.297777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.863950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6157.474774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6158.096021] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6158.704890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6159.286171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6159.913850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6160.498220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.129728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.725407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.342032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6163.017705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6163.724638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6164.454951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.151827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.687518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.301333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.855352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6167.422798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6167.988279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6168.605548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6169.154955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6169.756750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6170.420456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6171.033975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6171.642082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6172.267282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6172.873566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6173.566850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.179899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.847850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6175.441075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6176.053257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6176.755065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.377112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.933990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6178.515913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6179.088955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6179.700684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6180.267239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6180.889557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6181.454216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6182.060700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6182.631570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.213233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.798171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.397516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.997865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6185.571351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6186.112486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6186.650468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.185701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.740161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.327752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.908790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6189.471901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.095369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.667995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.247290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.831711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6192.419728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6192.965731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6193.571939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.234179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.841508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.416341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.001228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.596252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.262113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.820382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.429902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6199.044750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6199.697791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6200.387532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6201.017569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6201.616685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6202.251543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6202.827035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.421636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.980667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6204.569845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6205.226496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6205.812538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.411280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.996446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6207.579602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6208.185528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6208.773688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6209.404527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6209.998879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6210.625361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6211.178263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6211.774417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6212.393887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6213.012995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6213.619186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6214.239407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6214.841461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6215.462301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6216.036517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6216.594643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6217.129765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6217.675407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6218.249678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6218.861488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6219.461195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6220.074022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6220.698034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6221.325770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6221.922212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6222.530164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6223.091105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6223.707877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6224.369247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6225.061848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6225.633164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6226.266728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6226.829444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6227.482084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6228.074344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6228.719245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6229.300790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6229.925777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6230.555961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6231.164322] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6231.166304] Lustre: Skipped 215 previous similar messages [ 6231.175640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6231.179017] Lustre: Skipped 215 previous similar messages [ 6231.180409] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6231.182130] Lustre: Skipped 215 previous similar messages [ 6231.183512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6231.188056] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.206.107@tcp (at 0@lo) [ 6231.190746] Lustre: Skipped 215 previous similar messages [ 6231.754066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6232.317340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6232.853302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6233.429926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6233.951784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6234.553266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6235.135072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6235.734295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6236.329739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6236.920453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6237.429900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6237.947864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6238.465272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6239.032652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6239.573345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6240.162261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6240.682467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6241.257720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6241.828067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6242.368373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6242.904830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6243.460098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6244.002977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6244.559131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6245.109047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6245.628731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6246.153104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6246.709411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6247.331715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6247.939745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6248.535251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6249.186259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6249.767463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6250.391047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6250.944494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6251.514612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6252.057837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6252.611086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6253.198225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6253.809812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6254.367278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6254.969117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6255.535098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6256.121929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6256.661540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6257.200266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6257.727274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6258.300471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6258.895200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6259.475127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6260.053089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6260.721387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6261.320745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6261.914773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6262.455096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6262.995239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6263.557937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6264.144341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6264.757282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6265.399963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6266.025116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6266.654157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6267.238268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6267.818781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6268.394227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6268.994851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6269.558871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6270.169373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6270.791682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6271.382609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6271.922771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6272.458666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6272.999398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6273.544678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6274.109564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6274.709382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6275.302568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6275.933311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6276.506223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6277.100696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6277.642986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6278.222331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6278.783084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6279.318991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6279.890957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6280.516649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6281.096848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6281.691422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6282.331499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6282.910751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6283.484208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6284.101729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6284.660768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6285.247564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6285.855619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6286.482092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6287.093348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6287.733632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6288.384195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6289.046587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6289.626981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6290.174988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6290.735128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6291.344433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6291.904030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6292.496260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6293.062415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6293.629878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6294.239794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6294.869073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6295.463609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6296.115969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6296.687851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6297.287243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6297.877818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6298.497332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6299.152838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6299.847750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6300.500209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6301.112220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6301.701333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6302.314185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6302.947920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6303.604807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6304.180720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6304.769066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6305.313406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6305.891843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6306.515983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6307.115483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6307.723584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6308.310651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6308.856527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6309.426499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6310.058680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6310.643206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6311.227106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6311.886928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6312.493857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6313.114383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6313.708755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6314.327674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6314.892756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6315.529113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6316.140833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6316.775880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6317.356718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6317.972879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6318.549996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6319.139166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6319.688612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6320.265372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6320.834499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6321.463892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6322.073557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6322.643630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6323.168208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6323.675233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6324.404888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6324.955956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6325.536716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6326.125349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6326.763780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6327.371341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6327.941883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6328.484490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6329.034713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6329.584097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6330.205881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6330.783278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6331.393431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6332.028855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6332.674564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6333.229182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6333.805966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6334.335659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6334.913381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6335.463703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6336.079406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6336.616716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6337.183191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6337.754390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6338.317798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6338.859041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6339.418355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6339.952095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6340.519598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6341.065765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6341.655207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6342.247412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6342.858823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6343.430026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6344.021277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6344.587131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6345.168054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6345.760543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6346.352039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6346.912040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6347.535995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6348.130451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6348.727432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6349.306639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6349.882911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6350.448809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6351.077757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6351.652908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6352.252053] LustreError: 162361:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6352.256518] LustreError: 162361:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6352.262256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6352.879888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6353.546568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6354.116249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6354.665945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6355.199083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6355.761959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6356.336801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6356.891798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6357.440302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6358.028650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6358.608039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6359.219202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6359.796580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6360.379737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6360.932501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6361.501585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6362.056220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6362.618965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6363.164182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6363.728228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6364.273755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6364.843416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6365.407446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6366.010201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6366.553049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6367.163738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6367.744393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6368.353035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6368.932537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6369.522247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6370.063457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6370.691871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6371.282133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6371.894223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6372.425590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6373.010208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6373.566833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6374.121930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6374.662815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6375.207195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6375.762645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6376.429847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6377.019509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6377.632076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6378.230024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6378.833828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6379.459649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6380.038738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6380.617476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6381.206270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6381.768534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6382.418157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6382.987564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6383.637328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6384.346500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6384.938529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6385.552787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6386.111885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6386.749648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6387.325052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6387.884892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6388.518193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6389.140663] LustreError: 162361:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6389.144773] LustreError: 162361:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6389.151513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6389.749385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6390.332840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6390.897458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6391.521267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6392.096136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6392.676768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6393.258481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6393.857553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6394.466663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6395.062827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6395.616331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6396.221939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6396.840201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6397.460300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6398.028734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6398.597056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6399.154055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6399.748503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6400.375070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6400.998688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6401.575203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6402.151706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6402.694070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6403.248808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6403.820116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6404.442554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6405.031816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6405.679327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6406.314886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6406.959819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6407.521331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6408.140963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6408.724439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6409.339777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6409.912145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6410.508788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6411.082590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6411.685021] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6412.374074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6413.057705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6413.635683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6414.283616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6414.885838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6415.534563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6416.165189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6416.839829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6417.511355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6418.206186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6418.822478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6419.433160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6419.990936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6420.619381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6421.190662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6421.804992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6422.432218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6423.084683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6423.654629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6424.251620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6424.853412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6425.495835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6426.124433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6426.786775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6427.417586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6428.086293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6428.687038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6429.300913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6429.898215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6430.488622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6431.115281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6431.745488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6432.359669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6433.047472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6433.676443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6434.369990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6435.014628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6435.712685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6436.354350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6436.994250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6437.647543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6438.255527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6438.843236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6439.467591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6440.079725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6440.729164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6441.295622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6441.924556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6442.541243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6443.180632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6443.803392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6444.489955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6445.094981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6445.767668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6446.397644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6447.066930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6447.676682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6448.302854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6448.885938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6449.550140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6450.261592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6450.977837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6451.593445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6452.246786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6452.817218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6453.404860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6453.989363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6454.614133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6455.221410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6455.897638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6456.562084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6457.208854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6457.770875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6458.385700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6458.973879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6459.599507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6460.199961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6460.851457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6461.469754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6462.079949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6462.672027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6463.297885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6463.833607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6464.438836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6465.033016] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6465.650151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6466.235481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6466.856074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6467.489316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6468.135051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6468.772417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6469.366861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6469.954601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6470.661908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6471.299778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6471.988479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6472.554297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6473.160905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6473.727876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6474.377919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6474.988499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6475.583343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6476.142426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6476.774269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6477.383950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6477.977251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6478.530189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6479.112031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6479.632592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6480.183163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6480.779062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6481.371564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6481.930026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6482.462459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6483.012527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6483.574382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6484.068103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6484.615143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6485.181865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6485.793612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6486.372233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6486.935727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6487.504384] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6487.507404] Lustre: Skipped 430 previous similar messages [ 6487.515667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6487.519834] Lustre: Skipped 430 previous similar messages [ 6487.521846] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6487.525345] Lustre: Skipped 430 previous similar messages [ 6487.526795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6487.531540] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.206.107@tcp (at 0@lo) [ 6487.534608] Lustre: Skipped 430 previous similar messages [ 6488.054328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6488.579170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6489.188724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6489.748594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6490.370498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6490.947651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6491.546066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6492.140918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6492.759384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6493.328893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6493.928463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6494.548415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6495.222993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6495.801121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6496.385267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6497.028534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6497.714603] LustreError: 162361:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6497.718892] LustreError: 162361:0:(osp_precreate.c:654:osp_precreate_send()) Skipped 1 previous similar message [ 6497.722227] LustreError: 162361:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6497.726078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6497.727254] LustreError: 162361:0:(osp_precreate.c:1358:osp_precreate_thread()) Skipped 1 previous similar message [ 6498.342898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6498.960178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6499.575910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6500.212528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6500.811575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6501.464903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6502.102755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6502.803273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6503.442333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6504.104780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6504.739947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6505.393647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6505.988746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6506.680987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6507.308167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6507.954147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6508.545012] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6509.192422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6509.829361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6510.482058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6511.134858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6511.839429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6512.475539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6513.164743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6513.775730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6514.416688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6514.986182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6515.601723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6516.191747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6516.822211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6517.480902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6518.175807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6518.748241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6519.365546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6520.009528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6520.695570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6521.298244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6521.973823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6522.604234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6523.265567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6523.904549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6524.544873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6525.141497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6525.744665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6526.349624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6526.990877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6527.589203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6528.247389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6528.802106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6529.416029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6530.056132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6530.675188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6531.240344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6531.878175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6532.502985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6533.127826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6533.695047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6534.283301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6534.878467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6535.472800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6536.088521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6536.724703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6537.368415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6538.123179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6538.734265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6539.353675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6539.920624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6540.513940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6541.106429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6541.765753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6542.364181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6542.999252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6543.621894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6544.246958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6544.844034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6545.498097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6546.063241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6546.683600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6547.243638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6547.857605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6548.494923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6549.138258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6549.700778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6550.309653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6550.817850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6551.380055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6551.916642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6552.517542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6553.062317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6553.627908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6554.215957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6554.845876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6555.446233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6556.031909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6556.662528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6557.250660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6557.816240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6558.458742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6559.036590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6559.644651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6560.298179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6560.954098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6561.548800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6562.223337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6562.849849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6563.570777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6564.287610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6564.981300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6565.599550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6566.251071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6802.939679] LustreError: 163928:0:(ofd_dev.c:1668:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6802.942615] LustreError: 162361:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 6972.994555] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6972.997808] LustreError: Skipped 1 previous similar message [ 6972.999094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6973.924705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6973.926990] Lustre: Skipped 1 previous similar message [ 6979.042136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6979.045979] Lustre: Skipped 1 previous similar message [ 6984.163909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6987.232155] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6987.443757] Lustre: server umount lustre-OST0000 complete [ 6990.817015] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6990.819170] LustreError: Skipped 6 previous similar messages [ 6990.821236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6990.823087] Lustre: Skipped 1 previous similar message [ 6999.521301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6999.523741] Lustre: Skipped 3 previous similar messages [ 7004.128112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7004.347980] Lustre: server umount lustre-MDT0000 complete [ 7004.641272] LustreError: 163225:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7004.645323] LustreError: 163225:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 7006.682302] LustreError: 162321:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743502862 with bad export cookie 7883461711761661956 [ 7006.684942] LustreError: 162321:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7006.685896] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7006.862098] Lustre: server umount lustre-MDT0001 complete [ 7020.271391] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 7022.861119] Key type lgssc unregistered [ 7023.139684] LNet: 217082:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7024.166513] LNet: Removed LNI 192.168.206.107@tcp [ 7024.916030] Key type .llcrypt unregistered [ 7024.917167] Key type ._llcrypt unregistered [ 7049.988658] Key type ._llcrypt registered [ 7049.989632] Key type .llcrypt registered [ 7050.071997] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 7051.481838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7051.499145] alg: No test for adler32 (adler32-zlib) [ 7052.613721] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 7052.781939] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 7052.783741] LNet: Accept secure, port 988 [ 7054.416268] Key type lgssc registered [ 7055.255402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7060.152628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7060.163288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7061.435689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7063.666250] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7068.831636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7068.886888] Lustre: 219163:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7069.033610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7071.155747] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7075.229790] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7078.792813] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7083.634981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7083.763375] Lustre: 220045:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7083.887486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7085.935624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49666 to 0x280000401:49697) [ 7088.829356] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7092.662332] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7096.956226] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7098.071739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7098.077601] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7098.082063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7098.090592] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.206.107@tcp (at 0@lo) [ 7098.090622] LustreError: 218486:0:(osp_precreate.c:1266:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7099.362695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7099.363375] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7099.364954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7099.367832] Lustre: Skipped 1 previous similar message [ 7099.902810] Lustre: lustre-OST0000: Not available for connect from 192.168.206.7@tcp (stopping) [ 7099.905851] Lustre: Skipped 1 previous similar message [ 7104.483232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7104.485860] Lustre: Skipped 1 previous similar message [ 7105.482257] Lustre: server umount lustre-OST0000 complete [ 7118.168260] LustreError: 218486:0:(osp_precreate.c:1266:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7136.190434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7136.290229] Lustre: 221316:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7136.388547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7139.724071] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7144.014212] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7153.830544] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7153.835087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7153.835189] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.206.107@tcp (at 0@lo) [ 7153.841607] Lustre: Skipped 1 previous similar message [ 7157.276345] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 10 sec [ 7160.464207] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff926bd0a51000.ost_server_uuid 50 [ 7162.357338] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff926bd0a51000.ost_server_uuid in FULL state after 0 sec [ 7169.508087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7169.509401] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7169.509615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7169.509621] Lustre: Skipped 1 previous similar message [ 7169.513837] Lustre: Skipped 1 previous similar message [ 7171.199361] Lustre: server umount lustre-OST0000 complete [ 7174.625939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7174.626877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7174.635733] Lustre: Skipped 1 previous similar message [ 7174.637570] Lustre: Skipped 1 previous similar message [ 7179.745345] LustreError: 218467:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7179.752293] LustreError: 218467:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 7179.764255] Lustre: server umount lustre-MDT0000 complete [ 7182.130175] LustreError: 218446:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743503037 with bad export cookie 1179396158581992597 [ 7182.132935] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7182.133385] LustreError: 218446:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7182.318776] Lustre: server umount lustre-MDT0001 complete [ 7195.832869] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 7198.430036] Key type lgssc unregistered [ 7198.729613] LNet: 223129:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7199.782637] LNet: Removed LNI 192.168.206.107@tcp [ 7200.546686] Key type .llcrypt unregistered [ 7200.548176] Key type ._llcrypt unregistered [ 7232.494167] Key type ._llcrypt registered [ 7232.495367] Key type .llcrypt registered [ 7232.579594] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 7234.113389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7234.156345] alg: No test for adler32 (adler32-zlib) [ 7235.328545] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 7235.507209] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 7235.511375] LNet: Accept secure, port 988 [ 7237.160235] Key type lgssc registered [ 7237.976154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7244.115262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7244.129501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7245.446691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7247.729103] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7252.994615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7253.056777] Lustre: 225525:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7253.209976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7255.564672] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7257.502802] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 7261.626597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7261.729911] Lustre: 226262:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7261.826897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7265.252543] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7270.062707] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49699 to 0x280000401:49729) [ 7270.400285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7270.465950] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7270.609959] Lustre: 227057:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7270.642880] Lustre: lustre-OST0001: new disk, initializing [ 7270.650653] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7270.713741] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7271.220201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7271.223385] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7271.344297] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7274.293760] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7283.475600] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7290.231340] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 7304.673299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7304.676709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7304.683107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7307.236144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7307.239923] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7307.245631] Lustre: Skipped 2 previous similar messages [ 7310.289298] Lustre: server umount lustre-MDT0000 complete [ 7312.355932] LustreError: 224845:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7312.361790] LustreError: 224845:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 5 previous similar messages [ 7312.632171] LustreError: 224805:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743503167 with bad export cookie 4319311781338602205 [ 7312.636176] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7312.636632] LustreError: 224805:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7312.839817] Lustre: server umount lustre-MDT0001 complete [ 7325.721480] Lustre: server umount lustre-OST0000 complete [ 7338.411797] Lustre: server umount lustre-OST0001 complete [ 7340.559366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 7340.732748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7341.464264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 7341.544269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7342.331649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 7342.494652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7343.284770] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 7343.422080] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7356.658430] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 7364.363826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7364.655427] LustreError: 230157:0:(ldlm_lib.c:1095:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7364.706781] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7366.926142] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7370.209251] LustreError: 230157:0:(ldlm_lib.c:1095:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7372.195400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7372.241978] Lustre: 230857:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for mylustre-sptlrpc, check MGS config [ 7374.813281] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7376.625482] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 7380.770445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7380.887713] Lustre: 231604:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for mylustre-sptlrpc, check MGS config [ 7381.005341] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7381.007907] Lustre: Skipped 1 previous similar message [ 7382.055802] LustreError: 231611:0:(ldlm_lib.c:1095:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7384.627384] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7387.107621] LustreError: 231708:0:(ldlm_lib.c:1095:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7389.966492] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7390.018685] Lustre: 232401:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for mylustre-sptlrpc, check MGS config [ 7391.153183] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49732 to 0x280000401:49761) [ 7393.594966] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7400.562723] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7404.052996] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 7416.805663] Lustre: mylustre-MDT0000-lwp-OST0000: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7416.807298] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7416.809621] Lustre: Skipped 1 previous similar message [ 7416.812244] Lustre: Skipped 3 previous similar messages [ 7417.604273] Lustre: server umount mylustre-MDT0000 complete [ 7419.773936] LustreError: 230133:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743503275 with bad export cookie 4319311781338604970 [ 7419.778398] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7419.778710] LustreError: 230133:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7421.921868] LustreError: 230150:0:(ldlm_lib.c:1095:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7421.922712] Lustre: mylustre-MDT0001-lwp-OST0000: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7421.928212] LustreError: 230150:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 7421.933715] Lustre: Skipped 3 previous similar messages [ 7422.029370] Lustre: server umount mylustre-MDT0001 complete [ 7424.391909] Lustre: server umount mylustre-OST0000 complete [ 7426.641115] Lustre: server umount mylustre-OST0001 complete [ 7428.699052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 7428.825602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7429.536458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 7429.653038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7430.361771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 7430.450112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7431.178968] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 7431.276625] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7446.033690] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 7452.977761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7453.277067] LustreError: 235400:0:(ldlm_lib.c:1095:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7453.328068] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7453.330856] Lustre: Skipped 1 previous similar message [ 7455.800620] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7461.383477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7461.449359] Lustre: 236101:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for tfs-sptlrpc, check MGS config [ 7464.074993] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7465.978428] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 7470.162401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7473.957955] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7477.739471] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49763 to 0x280000401:49793) [ 7479.029684] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7482.488303] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7489.207337] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7492.936121] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 7502.817763] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7502.820060] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7502.824393] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 7507.832475] Lustre: server umount tfs-MDT0000 complete [ 7509.959741] LustreError: 235375:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743503365 with bad export cookie 4319311781338606636 [ 7509.964788] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7509.969459] LustreError: 235375:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7509.985434] LustreError: 238586:0:(ldlm_lib.c:1095:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7509.987421] Lustre: tfs-MDT0001: Not available for connect from 0@lo (stopping) [ 7509.990055] LustreError: 238586:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 6 previous similar messages [ 7509.992483] Lustre: Skipped 5 previous similar messages [ 7510.155805] Lustre: server umount tfs-MDT0001 complete [ 7518.668438] Lustre: server umount tfs-OST0000 complete [ 7520.843293] Lustre: server umount tfs-OST0001 complete [ 7522.892528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 7522.962556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7523.685442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 7523.795831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7524.496174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 7524.581215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7525.349418] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 7525.424908] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7539.467858] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 7546.395286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7546.711754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7546.713919] Lustre: Skipped 3 previous similar messages [ 7549.119282] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7554.279618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7554.333040] Lustre: 241285:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7554.336373] Lustre: 241285:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 2 previous similar messages [ 7556.722067] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7558.548169] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 7562.573985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7566.089708] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7571.169571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7572.403775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49763 to 0x280000401:49825) [ 7575.001721] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7581.832389] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7596.924101] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 06:30:52 (1743503452) [ 7610.849245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7610.853343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7610.859900] Lustre: Skipped 5 previous similar messages [ 7610.862874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7616.569387] Lustre: server umount lustre-MDT0000 complete [ 7618.533942] LustreError: 240584:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7618.539684] LustreError: 240584:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 7 previous similar messages [ 7618.759715] LustreError: 240559:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743503474 with bad export cookie 4319311781338608078 [ 7618.763430] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7618.764118] LustreError: 240559:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7628.770557] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7628.772052] Lustre: Skipped 7 previous similar messages [ 7633.376170] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7633.519893] Lustre: server umount lustre-MDT0001 complete [ 7650.272125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7650.320062] Lustre: server umount lustre-OST0000 complete [ 7658.756115] Lustre: server umount lustre-OST0001 complete [ 7662.712502] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 7672.312349] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 7681.919035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7686.839309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7691.319486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7696.591985] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7700.955607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7701.027489] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 7701.027489] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 7701.027489] [ 7701.038077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 7701.225413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7701.228447] Lustre: Skipped 1 previous similar message [ 7701.252596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7701.300610] Lustre: lustre-MDT0000: new disk, initializing [ 7701.353634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7701.355439] Lustre: Skipped 3 previous similar messages [ 7701.366904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7703.749491] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7710.411389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7710.467788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7710.554678] Lustre: 247427:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7710.559461] Lustre: 247427:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 2 previous similar messages [ 7710.580625] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7710.582276] Lustre: Skipped 1 previous similar message [ 7710.621581] Lustre: lustre-MDT0001: new disk, initializing [ 7710.704719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7710.708890] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7713.251463] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7719.870121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7719.923695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7720.104974] Lustre: lustre-OST0000: new disk, initializing [ 7720.108632] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7721.926532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7721.930707] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7722.007778] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7723.494883] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7730.170890] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7730.249358] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7730.367139] Lustre: lustre-OST0001: new disk, initializing [ 7730.371955] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7732.079666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7732.084358] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7732.212209] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7733.745907] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7745.354569] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 06:33:20 (1743503600) [ 7759.175120] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 06:33:34 (1743503614) [ 7773.157283] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7773.158683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7773.159981] LustreError: Skipped 1 previous similar message [ 7773.160690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7773.164449] Lustre: Skipped 6 previous similar messages [ 7773.170692] Lustre: Skipped 1 previous similar message [ 7778.638329] Lustre: server umount lustre-OST0000 complete [ 7782.369174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7783.395840] LustreError: 248196:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7783.402485] LustreError: 248196:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 4 previous similar messages [ 7787.122163] Lustre: server umount lustre-MDT0000 complete [ 7789.390449] LustreError: 246659:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743503644 with bad export cookie 4319311781338609408 [ 7789.395313] LustreError: 246659:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7789.395855] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7795.648765] Lustre: server umount lustre-MDT0001 complete [ 7811.903703] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 7819.078313] Lustre: server umount lustre-OST0001 complete [ 7823.048246] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 7838.498488] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 7840.897101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7846.886120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7852.068483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7856.849656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7861.842339] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7865.916248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7865.972891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7866.157586] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7866.159083] Lustre: Skipped 1 previous similar message [ 7866.179443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7866.223583] Lustre: lustre-MDT0000: new disk, initializing [ 7866.279736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7866.282434] Lustre: Skipped 3 previous similar messages [ 7866.296251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7868.691172] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7875.248410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7875.329584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7875.408702] Lustre: 254351:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7875.413876] Lustre: 254351:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 2 previous similar messages [ 7875.570708] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7877.921052] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7883.492727] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7886.815322] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7891.089623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7891.148920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7891.318219] Lustre: lustre-OST0000: new disk, initializing [ 7891.320168] Lustre: Skipped 1 previous similar message [ 7891.324988] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7891.327674] Lustre: Skipped 2 previous similar messages [ 7892.872378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7892.876039] Lustre: Skipped 1 previous similar message [ 7892.878642] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7892.970889] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7894.693700] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7900.144968] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7908.327191] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7908.327676] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7908.328599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7908.328605] Lustre: Skipped 8 previous similar messages [ 7908.330642] LustreError: Skipped 1 previous similar message [ 7908.335947] Lustre: Skipped 5 previous similar messages [ 7911.193921] Lustre: server umount lustre-OST0000 complete [ 7919.546253] Lustre: server umount lustre-MDT0000 complete [ 7921.824186] LustreError: 253581:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743503777 with bad export cookie 4319311781338610878 [ 7921.828122] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7921.828857] LustreError: 253581:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7922.052884] Lustre: server umount lustre-MDT0001 complete [ 7936.996752] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 7939.892414] Key type lgssc unregistered [ 7940.212826] LNet: 256965:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7941.222617] LNet: Removed LNI 192.168.206.107@tcp [ 7942.016746] Key type .llcrypt unregistered [ 7942.018195] Key type ._llcrypt unregistered [ 7968.946472] Key type ._llcrypt registered [ 7968.947296] Key type .llcrypt registered [ 7969.019100] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 7970.475523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7970.521663] alg: No test for adler32 (adler32-zlib) [ 7971.517153] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 7971.710670] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 7971.712569] LNet: Accept secure, port 988 [ 7973.344220] Key type lgssc registered [ 7974.090824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7978.721627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7978.732954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7980.027451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7982.400255] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7987.452884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7987.510650] Lustre: 259093:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7987.665328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7989.868221] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 7993.733533] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7997.022377] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8001.247055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8001.364923] Lustre: 259975:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8001.454433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8004.718363] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8005.549128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8008.787444] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8009.881832] Lustre: Setting parameter lustre-OST0000.ost.unknown_param in log lustre-OST0000 [ 8012.005951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8012.007088] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8016.036257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8016.039236] Lustre: Skipped 2 previous similar messages [ 8017.085100] Lustre: server umount lustre-OST0000 complete [ 8021.475630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8021.476829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8021.480970] Lustre: Skipped 1 previous similar message [ 8021.485271] Lustre: Skipped 1 previous similar message [ 8025.593441] Lustre: server umount lustre-MDT0000 complete [ 8026.594123] LustreError: 259102:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8026.600018] LustreError: 259102:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 8027.783857] LustreError: 258377:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743503883 with bad export cookie 307619041185465289 [ 8027.787745] LustreError: 258377:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8027.787929] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8027.983307] Lustre: server umount lustre-MDT0001 complete [ 8039.394867] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 8042.067176] Key type lgssc unregistered [ 8042.352970] LNet: 261609:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8043.366500] LNet: Removed LNI 192.168.206.107@tcp [ 8044.139555] Key type .llcrypt unregistered [ 8044.141132] Key type ._llcrypt unregistered [ 8059.910917] Key type ._llcrypt registered [ 8059.911772] Key type .llcrypt registered [ 8059.991111] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 8061.677825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8061.742373] alg: No test for adler32 (adler32-zlib) [ 8062.750720] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 8062.906413] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 8062.908575] LNet: Accept secure, port 988 [ 8064.544224] Key type lgssc registered [ 8065.355740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8070.822253] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8070.831804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8087.328472] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8093.472313] LustreError: 262773:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.107@tcp: failed processing log, type 4: rc = -110 [ 8123.232227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8126.245756] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8129.908282] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8131.449290] Lustre: server umount lustre-OST0000 complete [ 8145.621608] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 8148.102310] Key type lgssc unregistered [ 8148.376564] LNet: 264260:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8149.414387] LNet: Removed LNI 192.168.206.107@tcp [ 8150.103187] Key type .llcrypt unregistered [ 8150.104417] Key type ._llcrypt unregistered [ 8182.755161] Key type ._llcrypt registered [ 8182.757338] Key type .llcrypt registered [ 8182.836651] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 8184.617816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8184.641644] alg: No test for adler32 (adler32-zlib) [ 8185.879748] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 8186.106760] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 8186.109629] LNet: Accept secure, port 988 [ 8187.752182] Key type lgssc registered [ 8188.551366] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8196.510239] loop0: detected capacity change from 0 to 204800000 [ 8196.697911] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8196.701046] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8196.835400] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8196.840679] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8196.867124] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8196.890934] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8196.910920] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8196.948417] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8196.999140] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8197.041322] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8197.767349] loop0: detected capacity change from 0 to 204800000 [ 8198.009922] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8198.358807] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8199.025314] loop0: detected capacity change from 0 to 204800000 [ 8199.390057] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8199.970584] loop0: detected capacity change from 0 to 204800000 [ 8200.265378] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8200.900537] loop0: detected capacity change from 0 to 204800000 [ 8200.920225] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8203.331817] loop0: detected capacity change from 0 to 204800000 [ 8203.351646] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8205.811884] loop0: detected capacity change from 0 to 204800000 [ 8205.830794] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8208.262481] loop0: detected capacity change from 0 to 204800000 [ 8208.285361] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8210.764854] loop0: detected capacity change from 0 to 204800000 [ 8210.834728] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8210.943284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8210.957274] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8211.013190] Lustre: 267649:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8211.018901] Lustre: 267649:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 8211.024341] Lustre: 267649:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 8211.029765] Lustre: 267649:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8211.035541] Lustre: 267649:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8211.039324] Lustre: 267649:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8211.042487] Lustre: 267649:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 8211.045353] Lustre: 267649:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 8211.048542] Lustre: 267649:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 8211.052136] Lustre: 267649:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 8211.056505] Lustre: 267649:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 8211.060437] Lustre: 267649:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 8211.065942] Lustre: 267649:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 8211.070859] Lustre: 267649:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 8214.947789] Lustre: server umount MGS complete [ 8215.489475] loop0: detected capacity change from 0 to 204800000 [ 8215.573481] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8215.913911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8215.955943] Lustre: lustre-MDT0000: Aborting client recovery [ 8216.605946] loop1: detected capacity change from 0 to 204800000 [ 8216.713995] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8216.829103] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8216.889866] Lustre: 268083:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8216.894067] Lustre: 268083:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8216.898412] Lustre: 268083:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 8216.901856] Lustre: 268083:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8216.905506] Lustre: 268083:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 8216.909236] Lustre: 268083:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8216.912555] Lustre: 268083:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 8216.926384] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8216.932701] Lustre: 268083:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8217.060488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8217.111293] Lustre: lustre-MDT0001: Aborting client recovery [ 8217.654785] loop2: detected capacity change from 0 to 204800000 [ 8217.737479] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8217.834023] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8217.915053] Lustre: 268167:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8217.920453] Lustre: 268167:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8217.926490] Lustre: 268167:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8217.932361] Lustre: 268167:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8217.938115] Lustre: 268167:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8217.944394] Lustre: 268167:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8217.950142] Lustre: 268167:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8218.038647] Lustre: Found index 0 for lustre-OST0000, updating log [ 8218.045992] Lustre: 268167:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8218.123058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8218.137267] Lustre: lustre-OST0000: Aborting recovery [ 8218.791667] loop3: detected capacity change from 0 to 204800000 [ 8219.002615] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8219.130259] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8219.232278] Lustre: 268254:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8219.238459] Lustre: 268254:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8219.244500] Lustre: 268254:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8219.249790] Lustre: 268254:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8219.254591] Lustre: 268254:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8219.257444] Lustre: 268254:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8219.260488] Lustre: 268254:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8219.272728] Lustre: Found index 1 for lustre-OST0001, updating log [ 8219.278697] Lustre: 268254:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8219.372890] Lustre: lustre-OST0001: Aborting recovery [ 8224.753374] LustreError: 268026:0:(lod_dev.c:506:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -108 [ 8225.768432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8225.769199] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8225.793499] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 8225.795755] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 8226.791811] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 8226.792571] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 8230.885853] LustreError: 267994:0:(mdt_identity.c:95:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 309us: rc = -2 [ 8247.269262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8247.271427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8247.273848] Lustre: Skipped 2 previous similar messages [ 8247.277821] Lustre: Skipped 3 previous similar messages [ 8249.420741] Lustre: server umount lustre-MDT0000 complete [ 8250.031102] LustreError: 268092:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743504105 with bad export cookie 18271178702509099079 [ 8250.041474] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8252.385154] LustreError: 267990:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8252.385770] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8252.386619] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8252.389111] LustreError: 267990:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 8252.395762] Lustre: Skipped 2 previous similar messages [ 8255.392693] LustreError: 267990:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8255.392959] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8255.397348] LustreError: 267990:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 8255.399061] Lustre: Skipped 2 previous similar messages [ 8256.343430] Lustre: server umount lustre-MDT0001 complete [ 8271.328217] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8271.382892] Lustre: server umount lustre-OST0000 complete [ 8286.176150] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8286.299411] Lustre: server umount lustre-OST0001 complete [ 8294.766863] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 06:42:29 (1743504149) [ 8312.542993] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 8323.706661] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 8333.689392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8339.423293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8344.190686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8349.611792] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8354.017588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8354.063863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8354.219734] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8354.239831] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8354.278462] Lustre: lustre-MDT0000: new disk, initializing [ 8354.330825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8354.332447] Lustre: Skipped 1 previous similar message [ 8354.344050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8356.675409] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8363.299582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8363.354942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8363.439256] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8363.457390] Lustre: 273063:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8363.477869] LustreError: 272314:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8363.492754] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8363.495047] Lustre: Skipped 1 previous similar message [ 8363.542905] Lustre: lustre-MDT0001: new disk, initializing [ 8363.614790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8363.635403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8363.638250] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8365.913421] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8371.663809] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8375.352452] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8379.643494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8379.697942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8379.845592] Lustre: 274015:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8379.866426] Lustre: lustre-OST0000: new disk, initializing [ 8379.871078] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8379.921906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8380.897312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8380.902294] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8381.045862] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8383.580605] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8389.272936] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8393.862963] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 8402.079461] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 8408.143783] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 8416.844494] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 8462.818858] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8462.820451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8462.822810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8462.826337] Lustre: Skipped 1 previous similar message [ 8462.830688] Lustre: Skipped 1 previous similar message [ 8466.264248] Lustre: server umount lustre-OST0000 complete [ 8471.009238] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8471.012483] LustreError: Skipped 1 previous similar message [ 8471.014133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8471.018318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8474.745910] Lustre: server umount lustre-MDT0000 complete [ 8477.207365] LustreError: 272295:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743504332 with bad export cookie 18271178702509101788 [ 8477.211761] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8477.213791] LustreError: 272295:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8477.437904] Lustre: server umount lustre-MDT0001 complete [ 8483.271154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8485.873132] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8488.658537] Lustre: server umount MGS complete [ 8494.869154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8495.086267] LustreError: 277818:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8495.099305] LustreError: 277830:0:(qmt_pool.c:1513:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 8495.132764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8497.401373] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8500.193198] LustreError: 277818:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8502.362468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8502.426080] Lustre: 278516:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8504.875653] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8508.881773] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8512.094562] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8516.262452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8519.708912] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8523.965626] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8546.273100] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8546.275191] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8546.279098] Lustre: Skipped 1 previous similar message [ 8546.281367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8546.284154] Lustre: Skipped 2 previous similar messages [ 8549.690164] Lustre: server umount lustre-OST0000 complete [ 8562.657585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8562.660688] Lustre: Skipped 7 previous similar messages [ 8566.240180] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8566.340912] Lustre: server umount lustre-MDT0000 complete [ 8566.755172] LustreError: 277818:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8566.760163] LustreError: 277818:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 8568.585235] LustreError: 279424:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743504423 with bad export cookie 18271178702509103727 [ 8568.588776] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8568.590383] LustreError: 279424:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8568.798102] Lustre: server umount lustre-MDT0001 complete [ 8582.459277] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 8585.152350] Key type lgssc unregistered [ 8585.454691] LNet: 281405:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8586.470469] LNet: Removed LNI 192.168.206.107@tcp [ 8587.237443] Key type .llcrypt unregistered [ 8587.238963] Key type ._llcrypt unregistered [ 8612.737265] Key type ._llcrypt registered [ 8612.738623] Key type .llcrypt registered [ 8612.824641] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 8627.605882] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 8629.119535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8629.134697] alg: No test for adler32 (adler32-zlib) [ 8630.214284] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 8630.382671] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 8630.384864] LNet: Accept secure, port 988 [ 8632.016284] Key type lgssc registered [ 8632.793538] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8639.385145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8644.394716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8648.825970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8653.794229] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8657.924784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8657.970168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8657.979811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8659.145376] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8659.167094] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8659.211032] Lustre: lustre-MDT0000: new disk, initializing [ 8659.260950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8659.276430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8661.577948] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8668.327513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8668.383566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8668.469204] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8668.483826] Lustre: 286079:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8668.506639] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8668.508178] Lustre: Skipped 1 previous similar message [ 8668.546182] Lustre: lustre-MDT0001: new disk, initializing [ 8668.613773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8668.637961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8668.642124] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8670.960671] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8676.643606] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8680.230479] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8684.642719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8684.700619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8684.858695] Lustre: 287032:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8684.880605] Lustre: lustre-OST0000: new disk, initializing [ 8684.884169] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8684.950968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8688.357180] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8692.231968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8692.234607] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8692.311044] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8693.846384] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8699.330448] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 8710.415723] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 8719.115728] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 8730.554470] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 8763.875450] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8763.876664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8763.878955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8763.881658] Lustre: Skipped 1 previous similar message [ 8768.994429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8768.997068] Lustre: Skipped 2 previous similar messages [ 8774.116892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8774.119620] Lustre: Skipped 1 previous similar message [ 8778.208181] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8778.324404] Lustre: server umount lustre-OST0000 complete [ 8781.281218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8781.283724] LustreError: Skipped 1 previous similar message [ 8781.284911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8781.288786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8784.355726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8789.474397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8789.477497] Lustre: Skipped 3 previous similar messages [ 8795.104208] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8795.194524] Lustre: server umount lustre-MDT0000 complete [ 8797.766770] LustreError: 285309:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743504653 with bad export cookie 1006975916612105660 [ 8797.771865] LustreError: 285309:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8797.774248] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8797.986995] Lustre: server umount lustre-MDT0001 complete [ 8803.546338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8806.133071] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8809.194179] Lustre: server umount MGS complete [ 8816.049903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8816.329529] LustreError: 291061:0:(qmt_pool.c:1513:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 8816.369495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8818.681110] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8821.731928] LustreError: 291049:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8821.739093] LustreError: 291049:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 8823.885234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8823.939443] Lustre: 291746:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8824.101618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8826.263526] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8830.333600] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8834.074353] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8838.343215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8838.444138] Lustre: 292627:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8838.520546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8841.926214] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8845.929085] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8872.417871] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8872.419774] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8872.423333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8872.425027] Lustre: Skipped 2 previous similar messages [ 8873.419407] Lustre: server umount lustre-OST0000 complete [ 8879.586726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8879.591146] Lustre: Skipped 1 previous similar message [ 8889.824135] Lustre: 283536:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743504729/real 1743504729] req@000000000cefea99 x1828196959003008/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1743504745 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 8889.825709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8889.829291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8889.830778] Lustre: Skipped 5 previous similar messages [ 8890.336170] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8890.420363] Lustre: server umount lustre-MDT0000 complete [ 8892.654438] LustreError: 291029:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743504747 with bad export cookie 1006975916612107585 [ 8892.657346] LustreError: 291029:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8892.660610] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8892.869674] Lustre: server umount lustre-MDT0001 complete [ 8907.729506] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 8910.464620] Key type lgssc unregistered [ 8910.786530] LNet: 294639:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8911.846564] LNet: Removed LNI 192.168.206.107@tcp [ 8912.556929] Key type .llcrypt unregistered [ 8912.558566] Key type ._llcrypt unregistered [ 8928.286446] Key type ._llcrypt registered [ 8928.287994] Key type .llcrypt registered [ 8928.362421] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 8944.594861] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 8947.063159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8947.186409] alg: No test for adler32 (adler32-zlib) [ 8948.519128] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 8948.748031] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 8948.749670] LNet: Accept secure, port 988 [ 8950.392236] Key type lgssc registered [ 8951.299107] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8958.572830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8963.569708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8972.972589] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 8980.777966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8984.616125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8984.674434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8984.683748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8985.853835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8985.875185] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8985.912890] Lustre: lustre-MDT0000: new disk, initializing [ 8985.971447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8985.989156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8988.601575] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 8995.446736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8995.516109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8995.706222] Lustre: 298572:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8995.731871] Lustre: lustre-OST0000: new disk, initializing [ 8995.736519] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8995.739941] Lustre: Skipped 1 previous similar message [ 8995.789663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8999.377665] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9000.509830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9000.512563] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9000.522099] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9328.097803] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9328.105319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9332.182321] Lustre: server umount lustre-MDT0000 complete [ 9334.633765] LustreError: 298596:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743505189 with bad export cookie 17022773469915066014 [ 9334.639809] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9334.787159] Lustre: server umount lustre-OST0000 complete [ 9354.288536] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 9366.849362] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 9378.571495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9385.134512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9390.423392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9396.302852] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9400.709795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9400.767329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9400.968384] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9400.999496] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9401.058382] Lustre: lustre-MDT0000: new disk, initializing [ 9401.123148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9401.140324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9403.758779] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9410.390491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9410.441090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9410.507298] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9410.520833] Lustre: 303586:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9410.540168] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9410.542902] Lustre: Skipped 1 previous similar message [ 9410.591438] Lustre: lustre-MDT0001: new disk, initializing [ 9410.652668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9410.675409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9410.678032] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9412.905257] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9418.564420] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9421.685666] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9426.003364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9426.061746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9426.215494] Lustre: 304537:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9426.233052] Lustre: lustre-OST0000: new disk, initializing [ 9426.236638] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9426.288604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9429.577932] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9432.095493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9432.100169] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9432.212126] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9435.044842] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9437.657114] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9437.799925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9440.079990] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9440.254644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9442.274562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9442.275468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9442.278758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9442.280944] Lustre: Skipped 1 previous similar message [ 9447.428259] Lustre: server umount lustre-OST0000 complete [ 9451.489083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9451.492826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9451.497941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9451.499488] Lustre: Skipped 1 previous similar message [ 9455.729992] Lustre: server umount lustre-MDT0000 complete [ 9458.225384] LustreError: 302804:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743505313 with bad export cookie 17022773469918006805 [ 9458.231302] LustreError: 302804:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9458.231983] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9458.480154] Lustre: server umount lustre-MDT0001 complete [ 9461.259856] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [ 9463.267432] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 07:01:58 (1743505318) [ 9467.839470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9468.125754] LustreError: 306710:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9468.132975] LustreError: 306710:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 9468.171220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9470.504227] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9473.506329] LustreError: 306710:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9476.295418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9476.350772] Lustre: 307410:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9478.938529] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9483.558415] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9487.253449] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9491.400401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9491.491469] Lustre: 308290:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9491.567020] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9491.569980] Lustre: Skipped 1 previous similar message [ 9494.713740] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9498.927573] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9503.380548] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9503.441568] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9503.524855] Lustre: 309182:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9503.543751] Lustre: lustre-OST0001: new disk, initializing [ 9503.548132] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9507.096466] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9512.473567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9512.476447] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9512.515225] LustreError: 309865:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 9513.116255] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9513.290931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9513.571952] LustreError: 309865:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 9513.574667] LustreError: 309865:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 9514.599052] LustreError: 309865:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 9514.603797] LustreError: 309865:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 9516.194094] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff926bef9d9000.ost_server_uuid 50 [ 9516.644647] LustreError: 309865:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 9516.649223] LustreError: 309865:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 9517.839977] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff926bef9d9000.ost_server_uuid in FULL state after 0 sec [ 9520.745248] LustreError: 309874:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 9520.748146] LustreError: 309874:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [ 9521.127789] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff926bef9d9000.ost_server_uuid 50 [ 9522.856409] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff926bef9d9000.ost_server_uuid in FULL state after 0 sec [ 9523.843882] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9536.509568] Lustre: lustre-OST0001: Not available for connect from 192.168.206.7@tcp (stopping) [ 9536.511232] Lustre: Skipped 2 previous similar messages [ 9538.020178] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9538.025304] Lustre: Skipped 2 previous similar messages [ 9541.623132] Lustre: lustre-OST0001: Not available for connect from 192.168.206.7@tcp (stopping) [ 9541.626404] Lustre: Skipped 2 previous similar messages [ 9542.489978] Lustre: server umount lustre-OST0001 complete [ 9543.141452] LustreError: 310042:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9543.145822] LustreError: 310042:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 9547.233046] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9547.236226] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9548.261269] LustreError: 308297:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9548.267655] LustreError: 308297:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 9551.876395] Lustre: server umount lustre-OST0000 complete [ 9558.498099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9558.498378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9558.499212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9558.499217] Lustre: Skipped 3 previous similar messages [ 9558.501105] Lustre: Skipped 2 previous similar messages [ 9560.227198] Lustre: server umount lustre-MDT0000 complete [ 9562.419632] LustreError: 306737:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743505417 with bad export cookie 17022773469918007687 [ 9562.424108] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9562.425657] LustreError: 306737:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9562.643414] Lustre: server umount lustre-MDT0001 complete [ 9575.579071] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 9578.182121] Key type lgssc unregistered [ 9578.467563] LNet: 311631:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9579.494429] LNet: Removed LNI 192.168.206.107@tcp [ 9580.206768] Key type .llcrypt unregistered [ 9580.208295] Key type ._llcrypt unregistered [ 9604.311591] Key type ._llcrypt registered [ 9604.313224] Key type .llcrypt registered [ 9604.391579] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 9605.825816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9605.878311] alg: No test for adler32 (adler32-zlib) [ 9606.856419] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 9607.000965] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 9607.002967] LNet: Accept secure, port 988 [ 9608.624236] Key type lgssc registered [ 9609.317954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9613.837406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9613.850676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9615.133064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9617.173544] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9620.902283] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9625.111694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9625.159948] Lustre: 313816:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9625.320880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9627.388112] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9632.516482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9632.604839] Lustre: 314554:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9632.681081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9634.727123] LustreError: 314560:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9634.734107] LustreError: 314560:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 9635.833752] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9639.660865] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9639.909939] LustreError: 314560:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9640.935102] LustreError: 314560:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9640.940539] LustreError: 314560:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 9640.941468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 9643.917827] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9643.967098] Lustre: 315449:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9644.072485] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9647.228928] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9649.140984] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 9651.183632] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9655.408969] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 9655.559024] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9741.288533] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9741.290796] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9741.295176] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9741.296898] LustreError: Skipped 1 previous similar message [ 9743.752743] Lustre: server umount lustre-OST0001 complete [ 9746.404749] LustreError: 314749:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9746.411091] LustreError: 314749:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 9747.424986] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9747.427034] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9747.430609] Lustre: Skipped 1 previous similar message [ 9747.432460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9747.435383] Lustre: Skipped 1 previous similar message [ 9748.450105] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9748.453444] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9748.459387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9751.525908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9751.526377] LustreError: 314560:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9751.527468] Lustre: Skipped 1 previous similar message [ 9751.531419] LustreError: 314560:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 9753.535879] Lustre: server umount lustre-OST0000 complete [ 9756.643325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9756.650234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9756.652431] Lustre: Skipped 1 previous similar message [ 9762.173638] Lustre: server umount lustre-MDT0000 complete [ 9764.669328] LustreError: 316312:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743505620 with bad export cookie 6622083131037846377 [ 9764.672766] LustreError: 316312:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9764.673194] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9764.898281] Lustre: server umount lustre-MDT0001 complete [ 9780.164959] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 9783.413982] Key type lgssc unregistered [ 9783.744648] LNet: 317564:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9784.806606] LNet: Removed LNI 192.168.206.107@tcp [ 9785.717313] Key type .llcrypt unregistered [ 9785.718889] Key type ._llcrypt unregistered [ 9827.169102] Key type ._llcrypt registered [ 9827.170723] Key type .llcrypt registered [ 9827.253630] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 9828.915834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9828.979164] alg: No test for adler32 (adler32-zlib) [ 9830.101914] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 9830.271382] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 9830.273321] LNet: Accept secure, port 988 [ 9831.936188] Key type lgssc registered [ 9832.932175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9838.985156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9838.997853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9840.306202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9842.874928] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9848.458851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9848.520246] Lustre: 320519:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9848.703120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9851.147372] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9853.068584] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 9857.248346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9857.362046] Lustre: 321263:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9857.464716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9858.475592] LustreError: 321270:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9858.480488] LustreError: 321270:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 9860.902884] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9863.656404] LustreError: 321373:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9864.677696] LustreError: 321270:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9864.685504] LustreError: 321270:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 9864.688902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 9866.608733] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9866.659250] Lustre: 322061:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9866.781429] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9869.537351] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 9870.358274] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9878.839340] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9886.994671] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 9902.564617] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9902.566523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9902.569479] Lustre: Skipped 2 previous similar messages [ 9902.572178] Lustre: Skipped 1 previous similar message [ 9907.683977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9907.686130] Lustre: Skipped 5 previous similar messages [ 9908.681316] Lustre: server umount lustre-MDT0000 complete [ 9910.944637] LustreError: 320537:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743505766 with bad export cookie 13223014980655005637 [ 9910.948967] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9910.949409] LustreError: 320537:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9911.124848] Lustre: server umount lustre-MDT0001 complete [ 9923.921759] Lustre: server umount lustre-OST0000 complete [ 9936.453103] Lustre: server umount lustre-OST0001 complete [ 9953.571690] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 9960.393686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9960.605743] LustreError: 324978:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9960.645367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9962.705962] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9966.049102] LustreError: 324978:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9967.302570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9967.349090] Lustre: 325678:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9969.745815] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9971.431179] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 9975.223949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9975.318271] Lustre: 326425:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9975.417157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9975.420024] Lustre: Skipped 1 previous similar message [ 9977.447834] LustreError: 326631:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9978.388479] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9983.106974] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9985.969513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 9985.969837] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 9986.241672] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 9992.647154] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9996.037467] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [10003.426141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10003.428180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10003.431666] Lustre: Skipped 1 previous similar message [10003.433480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10006.497783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10006.501297] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10006.504658] Lustre: Skipped 2 previous similar messages [10009.279298] Lustre: server umount lustre-MDT0000 complete [10011.400099] LustreError: 325698:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743505866 with bad export cookie 13223014980655007338 [10011.403354] LustreError: 325698:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [10011.403781] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10011.605083] Lustre: server umount lustre-MDT0001 complete [10024.266549] Lustre: server umount lustre-OST0000 complete [10035.868220] Lustre: server umount lustre-OST0001 complete [10038.846984] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [10040.808807] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [10042.707784] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 07:11:37 (1743505897) [10059.698804] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [10066.091441] loop0: detected capacity change from 0 to 1099511627776 [10066.406700] SGI XFS with ACLs, security attributes, quota, no debug enabled [10066.414815] *** VALIDATE xfs *** [10066.423546] XFS (loop0): Mounting V5 Filesystem [10066.433974] XFS (loop0): Ending clean mount [10067.368865] loop1: detected capacity change from 0 to 18691697672192 [10092.628806] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10103.611826] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [10112.672962] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [10121.038128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10125.800103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10130.076706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10134.514935] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10138.981459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10139.047851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10139.223636] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10139.249561] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10139.293895] Lustre: lustre-MDT0000: new disk, initializing [10139.346979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10139.350310] Lustre: Skipped 1 previous similar message [10139.366226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10141.785209] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10148.786310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10148.853290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10148.928714] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10148.945419] Lustre: 333730:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [10148.951485] Lustre: 333730:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [10148.975969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10148.978757] Lustre: Skipped 1 previous similar message [10149.018199] Lustre: lustre-MDT0001: new disk, initializing [10149.107560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10149.110901] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10151.677433] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10157.475096] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10161.088814] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10165.593966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10165.668823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10165.850715] Lustre: 334680:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [10165.872795] Lustre: lustre-OST0000: new disk, initializing [10165.876282] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10167.009416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10167.012969] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10167.132441] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10169.452551] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10175.033458] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10177.890878] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10178.063663] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10180.484146] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10180.655487] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10182.629591] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10182.630796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10182.632676] LustreError: Skipped 1 previous similar message [10182.634373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10182.635554] Lustre: Skipped 1 previous similar message [10182.642155] Lustre: Skipped 4 previous similar messages [10187.961733] Lustre: server umount lustre-OST0000 complete [10190.305639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10190.313256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10192.866263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10192.867953] Lustre: Skipped 3 previous similar messages [10196.416725] Lustre: server umount lustre-MDT0000 complete [10197.985366] LustreError: 333856:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10197.989618] LustreError: 333856:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [10198.814835] LustreError: 332962:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743506054 with bad export cookie 13223014980655008892 [10198.819040] LustreError: 332962:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [10198.820382] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10199.026157] Lustre: server umount lustre-MDT0001 complete [10206.989335] XFS (loop0): Unmounting Filesystem [10210.892796] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 07:14:25 (1743506065) [10215.771510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10216.050111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10216.051985] Lustre: Skipped 2 previous similar messages [10218.321253] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10223.461995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10223.519554] Lustre: 337797:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [10226.004589] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10230.065629] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10233.847167] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10238.719695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10242.726144] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10246.793454] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10256.353192] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10256.356872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10256.360598] Lustre: Skipped 1 previous similar message [10256.363427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10260.286362] Lustre: server umount lustre-OST0000 complete [10268.854601] Lustre: server umount lustre-MDT0000 complete [10271.085861] LustreError: 338707:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743506126 with bad export cookie 13223014980655009774 [10271.089984] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10271.091452] LustreError: 338707:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [10271.285055] Lustre: server umount lustre-MDT0001 complete [10286.274971] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [10289.100733] Key type lgssc unregistered [10289.413586] LNet: 340373:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10290.470448] LNet: Removed LNI 192.168.206.107@tcp [10291.241047] Key type .llcrypt unregistered [10291.242100] Key type ._llcrypt unregistered [10318.517180] Key type ._llcrypt registered [10318.518258] Key type .llcrypt registered [10318.602861] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [10320.207321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10320.258672] alg: No test for adler32 (adler32-zlib) [10321.398763] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [10321.671157] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [10321.673330] LNet: Accept secure, port 988 [10323.304144] Key type lgssc registered [10324.057600] Lustre: Echo OBD driver; http://www.lustre.org/ [10329.073404] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10329.085298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10330.379326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10332.866545] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10338.046225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10338.110916] Lustre: 342606:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [10338.282668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10340.920495] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10345.446473] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10349.790645] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10355.012857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10355.146302] Lustre: 343489:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [10355.253923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10359.196502] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10363.385832] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10370.555564] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.7@tcp (stopping) [10371.040980] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10371.042962] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10373.602886] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10373.603044] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10373.604875] Lustre: Skipped 1 previous similar message [10375.668215] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.7@tcp (stopping) [10375.789205] Lustre: server umount lustre-MDT0001 complete [10377.711610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10378.723173] LustreError: 341909:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10378.727146] LustreError: 341909:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [10380.789150] LustreError: 343592:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10382.918142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10382.969957] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10382.973121] Lustre: Found index 1 for lustre-MDT0001, updating log [10383.002203] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10383.032271] Lustre: 344739:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [10383.236165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10385.820385] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10388.458393] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.206.107@tcp (at 0@lo) [10389.993715] LustreError: 345158:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [10389.998267] LustreError: 345158:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [10390.003362] LustreError: 341901:0:(mgc_request.c:616:do_requeue()) failed processing log: -17 [10390.031178] LustreError: 345162:0:(obd_config.c:1995:class_config_llog_handler()) MGC192.168.206.107@tcp: cfg command failed: rc = -17 [10390.033869] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [10390.033869] [10690.531951] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10690.532962] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10690.536172] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10690.539403] Lustre: Skipped 1 previous similar message [10690.543744] Lustre: Skipped 1 previous similar message [10695.328817] Lustre: server umount lustre-MDT0001 complete [10695.650859] LustreError: 341910:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10695.654857] LustreError: 341910:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [10697.215743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10700.770646] LustreError: 341910:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10700.778209] LustreError: 341910:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [10702.427544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10702.484330] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10702.485946] Lustre: Found index 1 for lustre-MDT0001, updating log [10702.512929] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10702.537763] Lustre: 355352:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [10702.707431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10705.132628] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [10707.945865] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.206.107@tcp (at 0@lo) [10712.044689] LustreError: 355912:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [10712.047123] LustreError: 355912:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [10712.050246] LustreError: 341901:0:(mgc_request.c:616:do_requeue()) failed processing log: -17 [10712.053646] LustreError: 341901:0:(mgc_request.c:616:do_requeue()) Skipped 1 previous similar message [10712.086942] LustreError: 355921:0:(obd_config.c:686:class_setup()) Device 8 already setup (type osp) [10712.090806] LustreError: 355921:0:(obd_config.c:1995:class_config_llog_handler()) MGC192.168.206.107@tcp: cfg command failed: rc = -17 [10712.095599] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.206.107@tcp [10712.095599] [11008.509259] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.7@tcp (stopping) [11008.512553] Lustre: Skipped 1 previous similar message [11010.019192] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11010.020315] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11014.437108] Lustre: server umount lustre-MDT0001 complete [11015.138314] LustreError: 341910:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11015.142953] LustreError: 341910:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [11016.346814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11021.349242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11021.407728] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11021.409424] Lustre: Found index 1 for lustre-MDT0001, updating log [11021.434252] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [11021.461456] Lustre: 366011:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [11021.664797] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [11024.149855] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11026.925718] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.206.107@tcp (at 0@lo) [11028.458150] LustreError: 366479:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11028.460263] LustreError: 366479:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11028.462795] LustreError: 341901:0:(mgc_request.c:616:do_requeue()) failed processing log: -17 [11028.465878] LustreError: 341901:0:(mgc_request.c:616:do_requeue()) Skipped 1 previous similar message [11028.489061] LustreError: 366482:0:(lod_lov.c:211:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [11028.494206] LustreError: 366482:0:(obd_config.c:1995:class_config_llog_handler()) MGC192.168.206.107@tcp: cfg command failed: rc = -114 [11028.496518] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [11028.496518] [11344.355716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11344.358346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11344.358921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11344.360690] Lustre: Skipped 3 previous similar messages [11344.363422] Lustre: Skipped 4 previous similar messages [11347.757525] Lustre: server umount lustre-MDT0000 complete [11349.476034] LustreError: 343592:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11349.480392] LustreError: 343592:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 5 previous similar messages [11350.129451] LustreError: 341889:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743507205 with bad export cookie 13872732888781380373 [11350.133690] LustreError: 341889:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [11350.134081] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11354.593986] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11356.415785] Lustre: server umount lustre-MDT0001 complete [11359.103578] Lustre: server umount lustre-OST0000 complete [11364.749330] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [11374.653146] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [11384.027766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11389.758560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11395.280317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11400.562547] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11409.427397] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 07:34:24 (1743507264) [11413.913833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11413.967145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11414.135073] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [11414.158600] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11414.208536] Lustre: lustre-MDT0000: new disk, initializing [11414.264537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11414.277386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11416.764834] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11423.363912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11423.421166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11423.506884] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [11423.522385] Lustre: 379920:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [11423.546571] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11423.548312] Lustre: Skipped 1 previous similar message [11423.596394] Lustre: lustre-MDT0001: new disk, initializing [11423.671318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11423.694046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11423.699117] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11426.042443] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11431.889476] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11435.342672] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11440.208691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11440.278233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11440.430918] Lustre: 380872:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [11440.449980] Lustre: lustre-OST0000: new disk, initializing [11440.454454] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11440.506303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11444.057058] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11449.901338] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11450.374773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11450.378556] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11450.446673] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11459.553833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11459.555950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11459.560775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11459.562980] Lustre: Skipped 2 previous similar messages [11462.389322] Lustre: server umount lustre-MDT0000 complete [11464.713601] LustreError: 379152:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743507320 with bad export cookie 13872732888781382340 [11464.716805] LustreError: 379949:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [11464.718503] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11464.719229] LustreError: 379152:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [11464.903494] Lustre: server umount lustre-MDT0001 complete [11477.790985] Lustre: server umount lustre-OST0000 complete [11490.474812] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 07:35:45 (1743507345) [11502.888399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11503.180704] LustreError: 383520:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11503.187397] LustreError: 383520:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [11503.225271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11505.809690] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11511.191670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11511.246116] Lustre: 384218:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [11511.412196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11513.729803] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11517.843343] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11521.364670] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11523.364486] Lustre: Failing over lustre-MDT0000 [11523.371218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11523.372860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11523.374685] Lustre: Skipped 3 previous similar messages [11523.377671] Lustre: Skipped 4 previous similar messages [11523.489779] Lustre: server umount lustre-MDT0000 complete [11535.841719] LustreError: 383538:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11535.847039] LustreError: 383538:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 6 previous similar messages [11539.797771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11539.883792] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11540.030622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11540.063344] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11542.420775] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11545.058218] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [11545.062120] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.206.107@tcp (at 0@lo) [11545.072967] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [11548.915816] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [11555.298983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11558.582349] Lustre: server umount lustre-MDT0000 complete [11560.880599] LustreError: 383501:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743507416 with bad export cookie 13872732888781384272 [11560.883525] LustreError: 383501:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [11560.884470] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11561.060732] Lustre: server umount lustre-MDT0001 complete [11566.644503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11569.516304] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11574.966429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11575.171701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11575.175136] Lustre: Skipped 1 previous similar message [11577.600251] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11582.310806] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11585.971068] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11587.941705] Lustre: Failing over lustre-MDT0000 [11587.949202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11587.950601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11587.953653] Lustre: Skipped 3 previous similar messages [11587.956462] Lustre: Skipped 4 previous similar messages [11588.067297] Lustre: server umount lustre-MDT0000 complete [11600.354120] LustreError: 386501:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11600.358968] LustreError: 386501:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 7 previous similar messages [11604.464156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11604.540875] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11604.740413] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11607.302754] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11610.082721] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [11610.086610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.206.107@tcp (at 0@lo) [11610.090860] Lustre: Skipped 1 previous similar message [11610.097652] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [11613.549583] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [11623.084237] Lustre: server umount lustre-MDT0000 complete [11625.536675] LustreError: 386481:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743507480 with bad export cookie 13872732888781385203 [11625.541179] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11625.542631] LustreError: 386481:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [11625.728533] Lustre: server umount lustre-MDT0001 complete [11634.213623] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 07:38:09 (1743507489) [11644.661990] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [11654.543416] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [11664.687813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11670.065470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11674.589029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11679.577122] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11691.475073] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [11700.522795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11700.585979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11700.770730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [11700.791159] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11700.829698] Lustre: lustre-MDT0000: new disk, initializing [11700.885454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11700.889040] Lustre: Skipped 1 previous similar message [11700.904215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11703.253779] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11709.946604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11709.994690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11710.096212] Lustre: 393232:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [11710.099511] Lustre: 393232:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [11710.122258] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11710.124041] Lustre: Skipped 1 previous similar message [11710.163031] Lustre: lustre-MDT0001: new disk, initializing [11710.242491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11710.245577] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11712.760411] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11718.328701] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11721.530577] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11726.020958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11726.083253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11726.250141] Lustre: lustre-OST0000: new disk, initializing [11726.253043] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11729.505314] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11732.490367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11732.493385] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11732.521018] Lustre: *** cfs_fail_loc=1e0, val=0*** [11732.554083] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11732.568212] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [11734.837156] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11740.467455] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [11752.933851] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11752.935615] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11752.938067] LustreError: Skipped 1 previous similar message [11752.941093] Lustre: Skipped 2 previous similar messages [11752.943922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11752.947960] Lustre: Skipped 4 previous similar messages [11754.139588] Lustre: server umount lustre-OST0000 complete [11762.678363] Lustre: server umount lustre-MDT0000 complete [11763.169368] LustreError: 393262:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11763.177229] LustreError: 393262:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [11764.901329] LustreError: 392464:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743507620 with bad export cookie 13872732888781385602 [11764.904684] LustreError: 392464:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [11764.905020] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11765.223116] Lustre: server umount lustre-MDT0001 complete [11781.508543] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [11784.691826] Key type lgssc unregistered [11784.999644] LNet: 396039:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11786.022413] LNet: Removed LNI 192.168.206.107@tcp [11786.755906] Key type .llcrypt unregistered [11786.756937] Key type ._llcrypt unregistered [11812.810240] Key type ._llcrypt registered [11812.812023] Key type .llcrypt registered [11812.885500] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [11814.354511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11814.449752] alg: No test for adler32 (adler32-zlib) [11815.556464] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [11815.737574] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [11815.740555] LNet: Accept secure, port 988 [11817.384131] Key type lgssc registered [11818.133498] Lustre: Echo OBD driver; http://www.lustre.org/ [11823.031772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11823.047624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11824.325624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11826.620336] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11832.059617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11832.109619] Lustre: 398124:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [11832.249746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11834.393166] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11838.603894] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11842.405700] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11847.045898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11847.150524] Lustre: 399007:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [11847.252497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11850.627510] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [11855.197791] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11857.519128] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [11872.903261] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 07:42:07 (1743507727) [11874.511157] Lustre: Modifying parameter general.jobid_name in log params [11875.672392] Lustre: Modifying parameter general.jobid_name in log params [11884.197786] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 07:42:18 (1743507738) [11894.074584] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 07:42:29 (1743507749) [11895.001461] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb in log lustre-client [11903.080182] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb in log lustre-client [11903.082238] Lustre: Skipped 2 previous similar messages [11905.136171] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 07:42:39 (1743507759) [11908.107297] Lustre: Setting parameter general.osc.*.max_dirty_mb in log params [11908.110284] Lustre: Skipped 3 previous similar messages [11923.323747] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc in log lustre-client [11923.327508] Lustre: Skipped 6 previous similar messages [11925.893583] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 07:43:00 (1743507780) [11928.105119] Lustre: *** cfs_fail_loc=131b, val=2*** [11930.341718] Lustre: *** cfs_fail_loc=131b, val=2*** [11939.704601] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 07:43:14 (1743507794) [11941.164446] Lustre: Modifying parameter general.jobid_name in log params [11941.166043] Lustre: Skipped 3 previous similar messages [11953.669993] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 07:43:28 (1743507808) [11965.412546] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11965.412630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11965.413717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11965.421195] Lustre: Skipped 1 previous similar message [11970.534318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11970.535912] Lustre: Skipped 2 previous similar messages [11975.651963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11975.653609] Lustre: Skipped 1 previous similar message [11977.184171] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11977.298830] Lustre: server umount lustre-OST0000 complete [11980.770799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11980.770879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11980.774772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11980.779668] Lustre: Skipped 1 previous similar message [11985.785936] Lustre: server umount lustre-MDT0000 complete [11985.890062] LustreError: 402694:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11985.894303] LustreError: 402694:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [11988.024766] LustreError: 397408:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743507843 with bad export cookie 15495757101891765462 [11988.028904] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11988.221978] Lustre: server umount lustre-MDT0001 complete [12003.614787] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [12006.232086] Key type lgssc unregistered [12006.533451] LNet: 404919:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12007.590437] LNet: Removed LNI 192.168.206.107@tcp [12008.372561] Key type .llcrypt unregistered [12008.373665] Key type ._llcrypt unregistered [12020.419389] Key type ._llcrypt registered [12020.420434] Key type .llcrypt registered [12020.503653] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [12039.560359] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [12042.005359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12042.100209] alg: No test for adler32 (adler32-zlib) [12043.394261] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [12043.729438] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [12043.732521] LNet: Accept secure, port 988 [12045.392231] Key type lgssc registered [12046.325561] Lustre: Echo OBD driver; http://www.lustre.org/ [12053.595591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12058.688575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12063.316630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12068.583774] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12073.030119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12073.085539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12073.093556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12074.287046] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [12074.309849] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12074.351591] Lustre: lustre-MDT0000: new disk, initializing [12074.409957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12074.425637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12076.722420] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12083.470563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12083.534610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12083.616612] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [12083.631615] Lustre: 408798:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [12083.653358] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12083.656300] Lustre: Skipped 1 previous similar message [12083.700929] Lustre: lustre-MDT0001: new disk, initializing [12083.773666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12083.797243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12083.802616] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12086.010361] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12092.067393] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12096.091452] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12100.979474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12101.064135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12101.274172] Lustre: 409749:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [12101.293232] Lustre: lustre-OST0000: new disk, initializing [12101.297821] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12101.374357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12104.825689] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12106.758480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12106.763157] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12106.827958] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12110.268791] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12113.342283] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12113.546571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12116.108912] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12116.313366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12122.083830] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12122.086578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12122.087633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12122.093023] Lustre: Skipped 1 previous similar message [12123.653965] Lustre: server umount lustre-OST0000 complete [12127.203734] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12127.204597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12127.208118] Lustre: Skipped 1 previous similar message [12127.211110] Lustre: Skipped 1 previous similar message [12132.160326] Lustre: server umount lustre-MDT0000 complete [12134.401407] LustreError: 409774:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743507989 with bad export cookie 16868308834014560309 [12134.405333] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12134.582281] Lustre: server umount lustre-MDT0001 complete [12143.858838] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 07:46:38 (1743507998) [12148.032965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12148.282970] LustreError: 412070:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12148.287974] LustreError: 412070:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [12148.318797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12150.456617] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12153.315136] LustreError: 412070:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12155.490731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12155.544321] Lustre: 412769:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [12155.678666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12158.030584] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12161.729476] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12165.367967] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12170.166089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12170.273486] Lustre: 413648:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [12170.371392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12173.470265] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12177.801888] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12186.444844] LustreError: 414246:0:(llog.c:425:llog_init_handle()) llog type is not specified! [12186.997979] Lustre: Modifying parameter general.timeout in log params [12189.482160] Lustre: Modifying parameter general.timeout in log params [12189.483600] Lustre: Skipped 4 previous similar messages [12193.791896] Lustre: Modifying parameter general.timeout in log params [12193.793253] Lustre: Skipped 8 previous similar messages [12201.890319] Lustre: Modifying parameter general.timeout in log params [12201.892521] Lustre: Skipped 15 previous similar messages [12218.190399] Lustre: Modifying parameter general.timeout in log params [12218.192043] Lustre: Skipped 31 previous similar messages [12250.410587] Lustre: Modifying parameter general.timeout in log params [12250.412081] Lustre: Skipped 64 previous similar messages [12263.656672] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 07:48:38 (1743508118) [12273.457952] Lustre: server umount lustre-MDT0000 complete [12273.634582] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12273.637066] LustreError: Skipped 1 previous similar message [12273.638310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12273.643169] LustreError: 412088:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12275.676150] LustreError: 421211:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743508131 with bad export cookie 16868308834014561191 [12275.679183] LustreError: 421211:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [12275.684560] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12277.730230] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12277.731580] LustreError: 414205:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12277.733665] Lustre: Skipped 1 previous similar message [12277.739870] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12277.741511] Lustre: Skipped 1 previous similar message [12277.890579] Lustre: server umount lustre-MDT0001 complete [12280.484732] Lustre: server umount lustre-OST0000 complete [12285.809886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12288.344584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12290.753382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12293.428704] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12297.851858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12298.037848] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12298.195251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12300.734751] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12306.240952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12306.292699] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12306.320490] Lustre: 424127:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [12308.769159] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12313.240112] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12316.244662] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12320.504870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12320.598433] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12320.622429] Lustre: 425024:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [12324.006874] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12328.323133] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12330.765830] Lustre: Modifying parameter general.jobid_var in log params [12330.768571] Lustre: Skipped 5 previous similar messages [12335.452727] Lustre: server umount lustre-OST0000 complete [12342.241202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12342.243201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12342.246669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12347.874741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12347.876543] Lustre: Skipped 2 previous similar messages [12347.921966] Lustre: server umount lustre-MDT0000 complete [12350.057670] LustreError: 423409:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743508205 with bad export cookie 16868308834014563438 [12350.060998] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12350.063210] LustreError: 423409:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 5 previous similar messages [12360.796977] Lustre: server umount lustre-MDT0001 complete [12375.840606] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [12378.516129] Key type lgssc unregistered [12378.804631] LNet: 426829:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12379.878445] LNet: Removed LNI 192.168.206.107@tcp [12380.676298] Key type .llcrypt unregistered [12380.677869] Key type ._llcrypt unregistered [12411.615314] Key type ._llcrypt registered [12411.616440] Key type .llcrypt registered [12411.687094] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [12413.337307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12413.352555] alg: No test for adler32 (adler32-zlib) [12414.401197] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [12414.574619] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [12414.576733] LNet: Accept secure, port 988 [12416.208188] Key type lgssc registered [12416.955101] Lustre: Echo OBD driver; http://www.lustre.org/ [12421.798317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12421.809176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12423.078303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12425.289602] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12430.454858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12430.514921] Lustre: 428914:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [12430.822851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12433.397438] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12437.709509] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12441.488228] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12445.929550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12446.037693] Lustre: 429797:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [12446.245648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12449.560875] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12453.574882] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12462.725703] Lustre: Modifying parameter general.jobid_var in log params [12469.683583] Lustre: server umount lustre-MDT0000 complete [12470.755606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12470.759723] LustreError: 428215:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12470.761800] Lustre: Skipped 2 previous similar messages [12470.765536] LustreError: 428215:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [12471.994484] LustreError: 430543:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743508327 with bad export cookie 17233824998621888052 [12471.997770] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12471.998335] LustreError: 430543:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [12472.188841] Lustre: server umount lustre-MDT0001 complete [12483.883114] Lustre: server umount lustre-OST0000 complete [12488.894139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12490.955520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12493.188077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12495.740409] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12499.791760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12499.958858] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12499.977833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [12500.120796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12502.603956] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12508.054606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12508.101855] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12508.117276] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [12508.131704] Lustre: 433461:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [12508.356140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12510.669559] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12514.696632] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12518.517025] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12523.335070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12523.452793] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12523.482117] Lustre: 434355:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [12523.661277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12527.219532] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12531.924611] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12534.018370] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [12541.923229] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12541.925182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12541.925720] LustreError: Skipped 1 previous similar message [12541.932422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12541.934795] Lustre: Skipped 1 previous similar message [12544.187754] Lustre: server umount lustre-OST0000 complete [12547.042638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12547.043502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12547.046039] Lustre: Skipped 2 previous similar messages [12552.161684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12552.163871] Lustre: Skipped 2 previous similar messages [12552.557755] Lustre: server umount lustre-MDT0000 complete [12554.743444] LustreError: 432742:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743508410 with bad export cookie 17233824998621888878 [12554.747461] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12554.749376] LustreError: 432742:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [12554.935588] Lustre: server umount lustre-MDT0001 complete [12572.797403] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [12575.443698] Key type lgssc unregistered [12575.754652] LNet: 436173:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12576.806507] LNet: Removed LNI 192.168.206.107@tcp [12577.547939] Key type .llcrypt unregistered [12577.549702] Key type ._llcrypt unregistered [12604.284338] Key type ._llcrypt registered [12604.285305] Key type .llcrypt registered [12604.367813] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [12605.874925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12605.890428] alg: No test for adler32 (adler32-zlib) [12606.986946] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [12607.165566] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [12607.167146] LNet: Accept secure, port 988 [12608.808173] Key type lgssc registered [12609.624128] Lustre: Echo OBD driver; http://www.lustre.org/ [12614.452483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12614.462906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12615.731075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12618.164840] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12623.743747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12623.796643] Lustre: 438258:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [12624.091321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12626.589570] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12630.995479] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12634.066308] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12638.343452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12638.444995] Lustre: 439143:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [12638.632960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12641.993621] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12645.908961] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12654.241797] Lustre: Modifying parameter general.jobid_var in log params [12655.260452] Lustre: Modifying parameter general.jobid_var in log params [12655.262263] Lustre: Skipped 1 previous similar message [12656.277556] Lustre: Modifying parameter general.jobid_var in log params [12656.279285] Lustre: Skipped 1 previous similar message [12658.682714] Lustre: Modifying parameter general.jobid_var in log params [12658.684251] Lustre: Skipped 4 previous similar messages [12662.710810] Lustre: Modifying parameter general.jobid_var in log params [12662.712518] Lustre: Skipped 7 previous similar messages [12670.869120] Lustre: Modifying parameter general.jobid_var in log params [12670.870557] Lustre: Skipped 16 previous similar messages [12687.017327] Lustre: Modifying parameter general.jobid_var in log params [12687.019049] Lustre: Skipped 33 previous similar messages [12712.796471] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 07:56:07 (1743508567) [12715.178987] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [12717.846119] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 07:56:12 (1743508572) [12720.614289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12720.615420] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12720.615727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12720.617167] Lustre: Skipped 1 previous similar message [12725.733574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12725.735248] Lustre: Skipped 2 previous similar messages [12726.792347] Lustre: server umount lustre-OST0000 complete [12730.851374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12730.852756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12730.854783] Lustre: Skipped 1 previous similar message [12735.097384] Lustre: server umount lustre-MDT0000 complete [12735.970196] LustreError: 437562:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12735.973840] LustreError: 437562:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [12737.209117] LustreError: 437544:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743508592 with bad export cookie 8825029404234114405 [12737.214376] LustreError: 437544:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [12737.214740] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12737.438813] Lustre: server umount lustre-MDT0001 complete [12749.693674] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [12752.750066] Key type lgssc unregistered [12753.079956] LNet: 446506:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12754.150624] LNet: Removed LNI 192.168.206.107@tcp [12755.014194] Key type .llcrypt unregistered [12755.015161] Key type ._llcrypt unregistered [12758.967490] Key type ._llcrypt registered [12758.968465] Key type .llcrypt registered [12759.063284] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [12761.942820] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules [12764.942440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12765.070142] alg: No test for adler32 (adler32-zlib) [12766.074690] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [12766.254669] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [12766.256361] LNet: Accept secure, port 988 [12767.912194] Key type lgssc registered [12769.092328] Lustre: Echo OBD driver; http://www.lustre.org/ [12770.032499] LustreError: 447671:0:(lu_object.c:1383:lu_context_key_register()) cfs_race id 60d sleeping [12787.921194] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [12791.270898] LustreError: 447671:0:(lu_object.c:1383:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [12797.306137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12797.316464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12798.680692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12801.204637] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12808.163052] LustreError: 448670:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12808.169614] LustreError: 448670:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [12810.511179] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 07:57:45 (1743508665) [12813.282042] LustreError: 448670:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12813.480715] LustreError: 448698:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [12813.484771] LustreError: 448698:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [12813.487292] LustreError: 448698:0:(lod_dev.c:506:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 0, failed: rc = -5 [12813.680565] Lustre: server umount lustre-MDT0000 complete [12830.215989] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [12833.322196] Key type lgssc unregistered [12833.651995] LNet: 450167:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12834.662511] LNet: Removed LNI 192.168.206.107@tcp [12835.507581] Key type .llcrypt unregistered [12835.508858] Key type ._llcrypt unregistered [12854.472906] Key type ._llcrypt registered [12854.474771] Key type .llcrypt registered [12854.572796] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [12857.197643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12857.235633] alg: No test for adler32 (adler32-zlib) [12858.529130] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [12858.742695] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [12858.744430] LNet: Accept secure, port 988 [12860.384183] Key type lgssc registered [12861.280644] Lustre: Echo OBD driver; http://www.lustre.org/ [12866.182742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12866.195839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12867.489543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12869.570377] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12874.508964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12874.563914] Lustre: 452061:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [12874.875470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12877.362305] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12881.267972] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12884.291900] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12888.373105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12888.500622] Lustre: 452944:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [12888.736260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12892.071522] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12895.776372] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12934.730954] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 07:59:50 (1743508790) [12936.325696] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [12937.997797] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 07:59:53 (1743508793) [12940.258617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12940.259956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12940.261754] Lustre: Skipped 2 previous similar messages [12940.264416] Lustre: Skipped 2 previous similar messages [12945.380402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12945.383080] Lustre: Skipped 2 previous similar messages [12946.349117] Lustre: server umount lustre-MDT0000 complete [12948.351291] LustreError: 451345:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743508803 with bad export cookie 242798589182711882 [12948.356789] LustreError: 451345:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [12948.356898] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12948.542542] Lustre: server umount lustre-MDT0001 complete [12961.093972] Lustre: server umount lustre-OST0000 complete [12967.700065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12967.983258] LustreError: 455278:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12967.988380] LustreError: 455278:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [12968.059594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12970.334517] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12973.538030] LustreError: 455278:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12973.541931] LustreError: 455278:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [12975.303644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12975.353493] Lustre: 455976:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [12975.657432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12978.075672] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [12982.306737] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12986.161789] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12991.190458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12995.214470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12995.295300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12995.419536] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [12995.424143] LustreError: 455267:0:(mgs_handler.c:493:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [12995.446281] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [12995.448942] LustreError: 457122:0:(tgt_mount.c:2266:server_fill_super()) Unable to start targets: -98 [12995.451057] LustreError: 457122:0:(tgt_mount.c:1761:server_put_super()) no obd lustre-OST0000 [12995.452735] LustreError: 457122:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [12995.518932] Lustre: server umount lustre-OST0000 complete [12995.520392] LustreError: 457122:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -98 [13000.761760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13000.864396] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [13000.867703] LustreError: 455267:0:(mgs_handler.c:493:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [13000.871355] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [13000.874206] LustreError: 457585:0:(tgt_mount.c:2266:server_fill_super()) Unable to start targets: -98 [13000.876349] LustreError: 457585:0:(tgt_mount.c:1761:server_put_super()) no obd lustre-OST0000 [13000.878074] LustreError: 457585:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [13000.934197] Lustre: server umount lustre-OST0000 complete [13000.935977] LustreError: 457585:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -98 [13002.473130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13006.539096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13006.646166] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [13006.647737] Lustre: Found index 0 for lustre-OST0000, updating log [13006.650089] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [13006.678379] Lustre: 458114:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [13006.698802] Lustre: lustre-OST0000: new disk, initializing [13006.702099] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13006.851689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13010.067447] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [13012.153165] Lustre: Failing over lustre-OST0000 [13012.234681] Lustre: server umount lustre-OST0000 complete [13016.545355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13016.549315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13016.556233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13018.594070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13018.594444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13020.397752] Lustre: server umount lustre-MDT0000 complete [13020.641626] LustreError: 455279:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13020.645858] LustreError: 455279:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [13022.369063] LustreError: 455259:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743508877 with bad export cookie 242798589182714626 [13022.372274] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13022.372996] LustreError: 455259:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [13022.566191] Lustre: server umount lustre-MDT0001 complete [13030.221937] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 08:01:25 (1743508885) [13046.997185] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [13054.837821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13055.074809] LustreError: 460456:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13055.122742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13057.156263] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [13060.578316] LustreError: 460456:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13061.808602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13061.861788] Lustre: 461152:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [13064.409163] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [13066.083696] Lustre: Setting parameter general.debug_raw_pointers in log params [13069.620795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13073.193732] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [13078.283216] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13078.343478] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13078.402761] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [13078.433473] Lustre: 462686:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [13078.436351] Lustre: 462686:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [13078.452171] Lustre: lustre-OST0001: new disk, initializing [13078.455028] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [13080.840535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13080.843574] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13080.926862] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13081.986564] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [13088.228272] Lustre: DEBUG MARKER: Using TIMEOUT=20 [13093.170806] Lustre: Setting parameter general.lod.*.mdt_hash in log params [13099.520285] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.7@tcp (stopping) [13099.522506] Lustre: Skipped 1 previous similar message [13101.025564] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13101.028465] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13105.383472] Lustre: server umount lustre-MDT0001 complete [13106.663141] LustreError: 462707:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13106.666902] LustreError: 462707:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [13107.149223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13111.234776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13111.284661] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13111.286558] Lustre: Found index 1 for lustre-MDT0001, updating log [13111.311084] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [13111.333083] Lustre: 464065:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [13111.678193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13111.681433] Lustre: Skipped 3 previous similar messages [13113.067968] LustreError: 464191:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [13113.070163] LustreError: 464191:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [13113.072675] LustreError: 460447:0:(mgc_request.c:616:do_requeue()) failed processing log: -17 [13113.128234] LustreError: 464205:0:(obd_config.c:1995:class_config_llog_handler()) MGC192.168.206.107@tcp: cfg command failed: rc = -17 [13113.130666] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [13113.130666] [13113.942806] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [13116.915372] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.206.107@tcp (at 0@lo) [13124.825221] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 08:02:59 (1743508979) [13144.859320] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [13158.716455] Lustre: Modifying parameter general.debug_raw_pointers in log params [13173.048667] Lustre: DEBUG MARKER: Using TIMEOUT=20 [13176.005791] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [13244.898784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13244.899349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13244.901419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13244.903331] Lustre: Skipped 4 previous similar messages [13244.904639] Lustre: Skipped 5 previous similar messages [13250.509224] Lustre: server umount lustre-MDT0000 complete [13252.538948] LustreError: 461547:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743509107 with bad export cookie 242798589182715536 [13252.542811] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13252.544021] LustreError: 461547:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [13252.756438] Lustre: server umount lustre-MDT0001 complete [13265.436547] Lustre: server umount lustre-OST0000 complete [13277.733993] Lustre: server umount lustre-OST0001 complete [13281.198063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [13287.678911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13288.430085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [13293.815046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [13301.315987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13302.075467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [13320.429289] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [13330.230600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13330.244421] Lustre: lustre-MDT0000: reset Object Index mappings [13330.607545] LustreError: 471246:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13330.611308] LustreError: 471246:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [13330.670693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13333.052507] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [13337.868710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13337.876576] Lustre: lustre-MDT0001: reset Object Index mappings [13337.977870] Lustre: 471945:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [13340.614762] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [13342.249377] Lustre: Modifying parameter general.debug_raw_pointers in log params [13345.755266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13348.214071] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [13349.406118] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [13354.096121] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13355.453959] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:386) [13355.454965] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [13355.456450] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [13357.792701] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [13364.968031] Lustre: DEBUG MARKER: Using TIMEOUT=20 [13369.191847] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [13402.508600] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 08:07:37 (1743509257) [13430.242412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13430.244391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13430.248186] Lustre: Skipped 1 previous similar message [13430.250451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13430.253017] Lustre: Skipped 5 previous similar messages [13432.293462] Lustre: lustre-OST0000: haven't heard from client 0aff4c09-def8-4ae6-8adc-c4e298aa7067 (at 192.168.206.7@tcp) in 32 seconds. I think it's dead, and I am evicting it. exp ffff9cdec6e12000, cur 1743509287 expire 1743509257 last 1743509255 [13434.664620] Lustre: server umount lustre-MDT0000 complete [13436.721684] LustreError: 471225:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743509292 with bad export cookie 242798589182762352 [13436.728156] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13436.982681] Lustre: server umount lustre-MDT0001 complete [13449.693602] Lustre: server umount lustre-OST0000 complete [13462.262929] Lustre: server umount lustre-OST0001 complete [13466.146021] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [13476.157794] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [13486.250257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13490.982112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13495.149476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13500.820419] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13504.930836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13504.988490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13505.163787] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [13505.185367] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13505.224665] Lustre: lustre-MDT0000: new disk, initializing [13505.277618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13505.279284] Lustre: Skipped 3 previous similar messages [13505.289909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13507.416639] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [13511.618549] Lustre: server umount lustre-MDT0000 complete [13513.099834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13516.648651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13516.700950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13516.851886] Lustre: Found index 0 for lustre-MDT0000, updating log [13519.061984] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [13521.966484] Lustre: server umount lustre-MDT0000 complete [13532.062244] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 08:09:46 (1743509386) [13534.182258] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [13536.668454] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 08:09:51 (1743509391) [13554.181940] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [13561.793266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13564.157265] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [13568.974350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13569.027639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13569.105547] Lustre: 481119:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [13569.108103] Lustre: 481119:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 2 previous similar messages [13569.127307] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13569.128711] Lustre: Skipped 1 previous similar message [13569.166904] Lustre: lustre-MDT0001: new disk, initializing [13569.250275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13569.254306] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13571.412543] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [13574.590383] Lustre: Modifying parameter general.debug_raw_pointers in log params [13574.592972] Lustre: Skipped 3 previous similar messages [13578.312237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13578.369264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13578.520694] Lustre: lustre-OST0000: new disk, initializing [13578.523525] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13579.665087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13579.667964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13579.772946] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13581.693244] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [13588.037473] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13588.100281] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13588.206201] Lustre: lustre-OST0001: new disk, initializing [13588.209121] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [13589.794223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13589.797042] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13589.873264] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13591.517602] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [13599.596425] Lustre: DEBUG MARKER: Using TIMEOUT=20 [13931.600791] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 08:16:26 (1743509786) [13946.606577] Lustre: server umount lustre-MDT0000 complete [13948.390812] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13948.392841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13948.393770] LustreError: 483768:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13948.393781] LustreError: 483768:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [13948.394459] Lustre: Skipped 6 previous similar messages [13948.417625] LustreError: 480407:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743509803 with bad export cookie 242798589182822027 [13948.420930] LustreError: 480407:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [13948.423442] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13948.702738] Lustre: server umount lustre-MDT0001 complete [13966.794953] Lustre: server umount lustre-OST0000 complete [13984.026801] Lustre: server umount lustre-OST0001 complete [13987.514810] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [14000.070352] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [14011.177932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14017.417571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14022.689889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14027.781311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14032.079896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14032.141802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14032.306915] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [14032.309144] Lustre: Skipped 1 previous similar message [14032.344803] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14032.389513] Lustre: lustre-MDT0000: new disk, initializing [14032.446005] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14032.448305] Lustre: Skipped 5 previous similar messages [14032.461195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14034.484813] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [14040.480083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14040.536237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14040.612660] Lustre: 492939:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [14040.615520] Lustre: 492939:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 2 previous similar messages [14040.755872] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14042.822038] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [14047.852627] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14050.869553] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14054.749824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14054.807925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14056.975763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14056.978016] Lustre: Skipped 1 previous similar message [14056.979940] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14057.084765] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14058.020599] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [14063.055263] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14069.144549] Lustre: lustre-MDD0000: changelog on [14070.646450] Lustre: lustre-MDD0001: changelog on [14205.350496] Lustre: 492183:0:(llog_cat.c:968:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [14230.235052] Lustre: lustre-MDD0001: changelog off [14231.917777] Lustre: lustre-MDD0000: changelog off [14236.132423] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14236.133747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14236.134431] LustreError: Skipped 1 previous similar message [14236.138533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14236.140046] Lustre: Skipped 5 previous similar messages [14241.160967] Lustre: server umount lustre-OST0000 complete [14245.857978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14249.526619] Lustre: server umount lustre-MDT0000 complete [14251.491043] LustreError: 492188:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14251.495095] LustreError: 492188:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 4 previous similar messages [14251.553984] LustreError: 496930:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743510106 with bad export cookie 242798589183625137 [14251.559673] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14251.765818] Lustre: server umount lustre-MDT0001 complete [14265.092840] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [14268.067614] Key type lgssc unregistered [14268.395958] LNet: 497678:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14269.418654] LNet: Removed LNI 192.168.206.107@tcp [14270.251262] Key type .llcrypt unregistered [14270.252209] Key type ._llcrypt unregistered [14283.342070] Key type ._llcrypt registered [14283.344167] Key type .llcrypt registered [14283.442738] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [14297.774620] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [14300.012808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14300.039840] alg: No test for adler32 (adler32-zlib) [14301.365136] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [14301.549624] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [14301.551781] LNet: Accept secure, port 988 [14303.176176] Key type lgssc registered [14304.065527] Lustre: Echo OBD driver; http://www.lustre.org/ [14310.565314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14315.958585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14320.855398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14326.288712] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14330.063703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14330.122209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14330.132158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14331.320266] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [14331.346843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14331.397260] Lustre: lustre-MDT0000: new disk, initializing [14331.446469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14331.460881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14333.488761] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [14339.989345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14340.042794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14340.111932] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [14340.125699] Lustre: 501602:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [14340.146032] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14340.147359] Lustre: Skipped 1 previous similar message [14340.189883] Lustre: lustre-MDT0001: new disk, initializing [14340.254423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14340.274246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14340.277159] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14342.441797] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [14348.116693] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14351.494612] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14355.686722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14355.736447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14355.883506] Lustre: 502556:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [14355.899760] Lustre: lustre-OST0000: new disk, initializing [14355.903164] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14355.952487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14358.893187] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [14364.780661] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14365.719093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14365.722883] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14365.790343] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14373.718031] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_module obdecho/obdecho [14374.815909] LustreError: 503448:0:(echo_client.c:670:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [14374.818727] LustreError: 503448:0:(obd_config.c:780:class_setup()) setup ec failed (-22) [14375.980509] LustreError: 503544:0:(echo_client.c:1879:echo_md_handler()) Can not resolve the path : rc = -2 [14386.147581] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14386.149253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14386.150318] Lustre: Skipped 2 previous similar messages [14386.151654] Lustre: Skipped 1 previous similar message [14390.835533] Lustre: server umount lustre-MDT0000 complete [14392.776840] LustreError: 500834:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743510248 with bad export cookie 12816441025406123044 [14392.781617] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14392.781841] LustreError: 500834:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [14392.948374] Lustre: server umount lustre-MDT0001 complete [14405.316924] Lustre: server umount lustre-OST0000 complete [14418.072073] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [14430.914734] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [14441.575306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14447.720787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14452.647251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14457.461644] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14461.560971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14461.617815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14461.783541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [14461.803287] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14461.844207] Lustre: lustre-MDT0000: new disk, initializing [14461.892868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14461.904496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14464.217175] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [14470.569159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14470.627815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14470.693433] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [14470.707318] Lustre: 508003:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [14470.736944] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14470.738986] Lustre: Skipped 1 previous similar message [14470.780080] Lustre: lustre-MDT0001: new disk, initializing [14470.840344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14470.862880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14470.865667] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14472.997773] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [14479.874093] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14484.795205] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14490.117916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14490.200798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14490.360740] Lustre: 508954:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [14490.383907] Lustre: lustre-OST0000: new disk, initializing [14490.388534] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14490.457367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14494.046406] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [14499.854822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14499.858357] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14499.987955] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14500.044051] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14502.764598] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14502.941472] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14505.524736] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14505.680923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14510.051850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14510.051894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14510.055438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14510.058454] Lustre: Skipped 2 previous similar messages [14515.176357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14515.180139] Lustre: Skipped 1 previous similar message [14520.292908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14520.296462] Lustre: Skipped 1 previous similar message [14521.312135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14521.412916] Lustre: server umount lustre-OST0000 complete [14525.411283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14525.413264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14525.417337] Lustre: Skipped 2 previous similar messages [14529.804429] Lustre: server umount lustre-MDT0000 complete [14530.529569] LustreError: 508024:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14530.534146] LustreError: 508024:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 5 previous similar messages [14532.216849] LustreError: 507233:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743510387 with bad export cookie 12816441025406124003 [14532.224178] LustreError: 507233:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [14532.224570] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14532.424324] Lustre: server umount lustre-MDT0001 complete [14535.746685] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 08:26:30 (1743510390) [14538.336562] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [14541.213429] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 08:26:35 (1743510395) [14548.241639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14548.579747] LustreError: 511227:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14548.642156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14551.950382] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [14554.082455] LustreError: 511227:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14558.179288] LustreError: 511227:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14559.349791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14559.420218] Lustre: 511924:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [14562.430123] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [14567.317355] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14571.330893] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14577.298375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14577.479387] Lustre: 512806:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [14581.446473] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [14585.827220] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14600.673522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14600.677096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14600.684224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14600.687186] Lustre: Skipped 1 previous similar message [14602.446075] Lustre: server umount lustre-MDT0000 complete [14603.234446] LustreError: 511226:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14604.992485] LustreError: 511208:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743510460 with bad export cookie 12816441025406124885 [14604.997212] LustreError: 511208:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [14604.998812] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14608.354474] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14608.357338] Lustre: Skipped 2 previous similar messages [14611.342140] Lustre: server umount lustre-MDT0001 complete [14616.403587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [14626.879433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14627.065000] LustreError: 514737:0:(llog.c:1515:llog_backup()) MGC192.168.206.107@tcp: failed to open log lustre-sptlrpc: rc = -108 [14627.067936] Lustre: 514737:0:(mgc_request_server.c:558:mgc_llog_local_copy()) MGC192.168.206.107@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [14627.187579] LustreError: 514744:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14627.197440] LustreError: 514744:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 5 previous similar messages [14643.699309] LustreError: 514761:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14643.704115] LustreError: 514761:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 12 previous similar messages [14678.515100] LustreError: 514759:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14678.522795] LustreError: 514759:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 28 previous similar messages [14688.224505] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [14688.227782] Lustre: 514737:0:(mgc_request.c:1858:mgc_process_log()) MGC192.168.206.107@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [14718.944365] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [14718.946034] Lustre: 511218:0:(mgc_request.c:1858:mgc_process_log()) MGC192.168.206.107@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [14728.748962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14728.750594] Lustre: Skipped 2 previous similar messages [14730.789563] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.206.107@tcp (at 0@lo) [14730.929784] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [14736.011214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14739.424346] LustreError: 511218:0:(mgc_request.c:616:do_requeue()) failed processing log: -5 [14744.051473] LustreError: 514761:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14744.056646] LustreError: 514761:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 44 previous similar messages [14750.177300] LustreError: 499059:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005453ac28 x1828202905456896/t0(0) o250->MGC192.168.206.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [14752.878955] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [14755.816256] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.206.107@tcp (at 0@lo) [14757.780195] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14760.257783] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [14763.931827] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14766.249703] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [14776.291740] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14776.292595] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14776.293286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14776.293291] Lustre: Skipped 4 previous similar messages [14776.294723] Lustre: Skipped 1 previous similar message [14777.950436] Lustre: server umount lustre-MDT0000 complete [14779.984372] LustreError: 511207:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743510635 with bad export cookie 12816441025406131367 [14779.988652] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14779.988930] LustreError: 511207:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [14780.153569] Lustre: server umount lustre-MDT0001 complete [14792.774639] Lustre: server umount lustre-OST0000 complete [14806.875135] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [14817.760198] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [14827.261940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14832.363718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14837.629099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14843.512447] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14848.343811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14848.414054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14848.588455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [14848.610449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14848.655120] Lustre: lustre-MDT0000: new disk, initializing [14848.722245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14848.724476] Lustre: Skipped 1 previous similar message [14848.736986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14851.322352] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [14858.304972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14858.391325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14858.458532] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [14858.475803] Lustre: 520310:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [14858.635635] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14860.999239] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [14866.278183] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14869.301894] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14873.459131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14873.549297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14877.171094] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [14880.273370] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14880.381147] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14882.356978] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14884.959259] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14885.145377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14887.534488] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14887.726782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14890.468856] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14890.469754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14890.472117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14890.473885] Lustre: Skipped 2 previous similar messages [14890.475565] Lustre: Skipped 2 previous similar messages [14903.264149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14903.364755] Lustre: server umount lustre-OST0000 complete [14911.598796] Lustre: server umount lustre-MDT0000 complete [14913.805490] LustreError: 519540:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743510769 with bad export cookie 12816441025406148181 [14913.807916] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14913.808858] LustreError: 519540:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [14914.005366] Lustre: server umount lustre-MDT0001 complete [14924.287428] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 08:32:59 (1743510779) [14929.419533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14929.753560] LustreError: 523535:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14929.760944] LustreError: 523535:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 7 previous similar messages [14932.291065] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [14937.553185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14937.612796] Lustre: 524230:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [14937.615595] Lustre: 524230:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [14940.497574] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [14945.207601] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14949.374612] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14954.224420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14957.686152] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [14962.264350] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14975.606856] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 08:33:50 (1743510830) [14981.091343] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14981.095635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14981.102160] Lustre: Skipped 2 previous similar messages [14981.104760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14981.107929] Lustre: Skipped 9 previous similar messages [14984.642462] Lustre: server umount lustre-OST0000 complete [15001.056144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15001.149683] Lustre: server umount lustre-MDT0000 complete [15003.234140] LustreError: 524246:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743510858 with bad export cookie 12816441025406149063 [15003.239514] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15003.441412] Lustre: server umount lustre-MDT0001 complete [15018.077617] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [15020.719997] Key type lgssc unregistered [15021.007440] LNet: 526909:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15022.054638] LNet: Removed LNI 192.168.206.107@tcp [15022.858128] Key type .llcrypt unregistered [15022.859436] Key type ._llcrypt unregistered [15048.479059] Key type ._llcrypt registered [15048.479994] Key type .llcrypt registered [15048.586461] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [15050.982160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15051.019284] alg: No test for adler32 (adler32-zlib) [15052.251997] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [15052.627724] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [15052.629769] LNet: Accept secure, port 988 [15054.360271] Key type lgssc registered [15055.482895] Lustre: Echo OBD driver; http://www.lustre.org/ [15060.929369] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15060.941090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15078.496544] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [15084.640195] LustreError: 528132:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.107@tcp: failed processing log, type 4: rc = -110 [15090.784756] LustreError: 528132:0:(llog_osd.c:249:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [15090.788245] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [15090.790180] LustreError: MGC192.168.206.107@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [15090.793469] LustreError: 528132:0:(tgt_mount.c:1523:server_start_targets()) failed to start server lustre-OST0000: -22 [15090.796570] LustreError: 528132:0:(tgt_mount.c:2266:server_fill_super()) Unable to start targets: -22 [15090.799276] LustreError: 528132:0:(tgt_mount.c:1761:server_put_super()) no obd lustre-OST0000 [15090.865330] Lustre: server umount lustre-OST0000 complete [15090.866505] LustreError: 528132:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -22 [15097.158740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15098.431363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15100.707713] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15106.112509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15106.178862] Lustre: 529329:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [15106.333628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15108.791254] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15113.195091] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15117.810355] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15123.056954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15123.166245] Lustre: 530212:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [15123.171468] LustreError: 530212:0:(llog_osd.c:249:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [15123.174274] Lustre: 530212:0:(mgc_request_server.c:539:mgc_llog_local_copy()) MGC192.168.206.107@tcp: invalid local config log lustre-OST0000: rc = -22 [15123.266271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15127.209972] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15133.011980] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15135.096302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15135.097980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15135.099363] Lustre: Skipped 2 previous similar messages [15135.101991] Lustre: Skipped 2 previous similar messages [15140.324715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15140.328053] Lustre: Skipped 2 previous similar messages [15141.297172] Lustre: server umount lustre-MDT0000 complete [15143.646185] LustreError: 530856:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743510998 with bad export cookie 16510509710752940278 [15143.649782] LustreError: 530856:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [15143.650169] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15143.824705] Lustre: server umount lustre-MDT0001 complete [15155.849665] Lustre: server umount lustre-OST0000 complete [15162.751590] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [15177.699865] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [15188.738545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15195.256072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15200.675334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15207.361980] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15212.961379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15213.032880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15213.256544] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [15213.295232] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15213.357274] Lustre: lustre-MDT0000: new disk, initializing [15213.447358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15213.467373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15216.444790] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15223.734299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15223.814955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15223.890943] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [15223.909616] Lustre: 534369:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [15223.933976] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15223.935758] Lustre: Skipped 1 previous similar message [15223.976775] Lustre: lustre-MDT0001: new disk, initializing [15224.069849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15224.100984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15224.107071] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15226.679297] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15232.519867] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15236.435585] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15241.311637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15241.383979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15241.555173] Lustre: 535319:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [15241.573962] Lustre: lustre-OST0000: new disk, initializing [15241.577209] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15241.643298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15245.699164] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15247.899207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15247.902868] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15247.989610] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15251.910282] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15254.886485] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15255.053270] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15257.340934] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15257.524833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15263.204636] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15263.208996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15263.211787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15263.215754] Lustre: Skipped 1 previous similar message [15263.224209] Lustre: Skipped 1 previous similar message [15264.755471] Lustre: server umount lustre-OST0000 complete [15268.324160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15268.326203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15268.331261] Lustre: Skipped 1 previous similar message [15273.441275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15273.442877] Lustre: Skipped 2 previous similar messages [15274.147094] Lustre: server umount lustre-MDT0000 complete [15277.397836] LustreError: 533600:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743511132 with bad export cookie 16510509710752940978 [15277.401924] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15277.404825] LustreError: 533600:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [15277.608211] Lustre: server umount lustre-MDT0001 complete [15290.518641] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 08:39:05 (1743511145) [15309.156459] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [15312.185679] Key type lgssc unregistered [15312.502913] LNet: 538098:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15313.574862] LNet: Removed LNI 192.168.206.107@tcp [15314.582841] Key type .llcrypt unregistered [15314.583957] Key type ._llcrypt unregistered [15336.441272] Key type ._llcrypt registered [15336.442309] Key type .llcrypt registered [15336.583529] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [15339.065455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15339.167139] alg: No test for adler32 (adler32-zlib) [15340.363521] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [15340.538905] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [15340.541365] LNet: Accept secure, port 988 [15342.200254] Key type lgssc registered [15343.313473] Lustre: Echo OBD driver; http://www.lustre.org/ [15349.741046] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15349.761728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15351.086072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15353.914875] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15359.863605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15359.932775] Lustre: 539990:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [15360.078866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15362.653280] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15367.727369] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15371.935790] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15377.520069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15377.638632] Lustre: 540922:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [15377.647729] Lustre: *** cfs_fail_loc=131e, val=0*** [15377.649502] LustreError: 540922:0:(llog.c:1555:llog_backup()) MGC192.168.206.107@tcp: failed to backup log lustre-OST0000: rc = -28 [15377.652697] Lustre: 540922:0:(mgc_request_server.c:536:mgc_llog_local_copy()) MGC192.168.206.107@tcp: can't backup local config lustre-OST0000: rc = -28 [15377.717300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15381.283303] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15385.682800] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15387.723634] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15387.726328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15387.726666] Lustre: Skipped 1 previous similar message [15392.738858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15392.740965] Lustre: Skipped 4 previous similar messages [15393.914278] Lustre: server umount lustre-MDT0000 complete [15396.136532] LustreError: 540947:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743511251 with bad export cookie 4201005048283290542 [15396.140212] LustreError: 540947:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [15396.141218] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15396.323952] Lustre: server umount lustre-MDT0001 complete [15407.894705] Lustre: server umount lustre-OST0000 complete [15413.177645] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [15423.217102] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [15432.640679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15438.790909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15444.607380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15450.898588] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15455.628157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15455.702061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15455.895954] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [15455.916498] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15455.962284] Lustre: lustre-MDT0000: new disk, initializing [15456.020454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15456.032694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15458.365253] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15465.134494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15465.212737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15465.282333] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [15465.299179] Lustre: 545059:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [15465.312435] Lustre: *** cfs_fail_loc=131e, val=0*** [15465.313561] Lustre: Skipped 1 previous similar message [15465.315084] LustreError: 545059:0:(llog.c:1555:llog_backup()) MGC192.168.206.107@tcp: failed to backup log lustre-MDT0001: rc = -28 [15465.317357] LustreError: 545059:0:(llog.c:1555:llog_backup()) Skipped 1 previous similar message [15465.319150] Lustre: 545059:0:(mgc_request_server.c:558:mgc_llog_local_copy()) MGC192.168.206.107@tcp: failed to copy new config lustre-MDT0001: rc = -28 [15465.329914] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15465.331450] Lustre: Skipped 1 previous similar message [15465.375303] Lustre: lustre-MDT0001: new disk, initializing [15465.451643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15465.474988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15465.478462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15468.164526] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15474.088589] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15478.308150] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15483.660974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15483.719372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15483.863284] Lustre: 546009:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [15483.871758] Lustre: *** cfs_fail_loc=131e, val=0*** [15483.872721] Lustre: Skipped 1 previous similar message [15483.873883] LustreError: 546009:0:(llog.c:1555:llog_backup()) MGC192.168.206.107@tcp: failed to backup log lustre-OST0000: rc = -28 [15483.877214] LustreError: 546009:0:(llog.c:1555:llog_backup()) Skipped 1 previous similar message [15483.879073] Lustre: 546009:0:(mgc_request_server.c:558:mgc_llog_local_copy()) MGC192.168.206.107@tcp: failed to copy new config lustre-OST0000: rc = -28 [15483.881750] Lustre: 546009:0:(mgc_request_server.c:558:mgc_llog_local_copy()) Skipped 1 previous similar message [15483.892268] Lustre: lustre-OST0000: new disk, initializing [15483.894979] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15483.942568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15487.304375] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15490.566345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15490.571854] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15490.658314] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15493.411676] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15496.636656] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15496.811810] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15499.478122] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15499.670144] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15505.889923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15505.891500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15505.893230] LustreError: Skipped 1 previous similar message [15505.894379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15505.895880] Lustre: Skipped 2 previous similar messages [15507.077508] Lustre: server umount lustre-OST0000 complete [15511.011766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15511.013221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15511.018716] Lustre: Skipped 1 previous similar message [15511.023101] Lustre: Skipped 1 previous similar message [15515.524276] Lustre: server umount lustre-MDT0000 complete [15516.129837] LustreError: 545089:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15516.134677] LustreError: 545089:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [15518.148677] LustreError: 544291:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743511373 with bad export cookie 4201005048283291256 [15518.152042] LustreError: 544291:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [15518.153312] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15518.353878] Lustre: server umount lustre-MDT0001 complete [15528.297218] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 08:43:03 (1743511383) [15545.372870] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [15554.205664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15554.456447] LustreError: 548975:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15554.486227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15557.067905] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15559.650708] LustreError: 548975:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15562.901388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15562.956297] Lustre: 549671:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [15565.364429] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15567.109742] Lustre: Modifying parameter general.debug_raw_pointers in log params [15571.169344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15571.281326] Lustre: 550408:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [15575.192545] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15580.413479] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15580.500946] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15580.639903] Lustre: lustre-OST0001: new disk, initializing [15580.643265] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [15581.580400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [15581.583495] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [15581.708979] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [15584.234973] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15593.872930] Lustre: DEBUG MARKER: Using TIMEOUT=20 [15605.357847] Lustre: Setting parameter general.lod.*.mdt_hash in log params [15612.002556] Lustre: DEBUG MARKER: ADD OST3 [15616.353968] loop0: detected capacity change from 0 to 204800000 [15616.624500] print_req_error: 14 callbacks suppressed [15616.624513] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15616.628816] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15616.632285] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15616.713530] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15620.782384] loop0: detected capacity change from 0 to 204800000 [15623.461791] Lustre: DEBUG MARKER: STOP OST3 [15625.756866] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15625.853186] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15626.012729] Lustre: 553070:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [15626.016257] Lustre: 553070:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [15626.045640] Lustre: lustre-OST0003: new disk, initializing [15626.053879] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [15626.223452] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [15626.225509] Lustre: Skipped 3 previous similar messages [15627.878738] Lustre: *** cfs_fail_loc=2109, val=2*** [15628.047761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [15628.051033] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [15628.086471] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [15629.076491] Lustre: Failing over lustre-OST0003 [15629.180422] Lustre: server umount lustre-OST0003 complete [15629.920572] LustreError: 553263:0:(osp_precreate.c:1283:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [15631.240571] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15632.903868] LustreError: 550593:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [15633.393551] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15633.395096] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15633.401453] Lustre: Skipped 1 previous similar message [15638.009789] LustreError: 550415:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [15638.013842] LustreError: 550415:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [15638.436456] Lustre: DEBUG MARKER: START OST3 again [15642.682843] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15642.882635] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [15643.125296] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [15644.542816] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 192.168.206.107@tcp (at 0@lo) [15644.543057] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [15646.959602] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15668.461131] Lustre: Failing over lustre-OST0003 [15668.517448] Lustre: server umount lustre-OST0003 complete [15668.723963] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15668.738984] LustreError: 550415:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15668.746235] LustreError: 550415:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [15684.069103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15684.071561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15684.072623] Lustre: Skipped 4 previous similar messages [15684.074316] Lustre: Skipped 2 previous similar messages [15689.194721] LustreError: 550593:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15689.202398] LustreError: 550593:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 5 previous similar messages [15694.311167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15694.314385] Lustre: Skipped 7 previous similar messages [15698.400144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15698.531441] Lustre: server umount lustre-MDT0000 complete [15700.595887] LustreError: 548954:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743511555 with bad export cookie 4201005048283292138 [15700.603560] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15704.547022] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15704.551455] Lustre: Skipped 1 previous similar message [15706.994058] Lustre: server umount lustre-MDT0001 complete [15710.263671] Lustre: server umount lustre-OST0000 complete [15713.250810] Lustre: server umount lustre-OST0001 complete [15718.290240] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [15729.538317] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [15739.546191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15747.024571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15751.754543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15756.629969] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15760.878620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15760.935339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15761.109351] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [15761.138390] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15761.183271] Lustre: lustre-MDT0000: new disk, initializing [15761.236318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15761.239859] Lustre: Skipped 1 previous similar message [15761.254582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15763.673533] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15772.081775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15772.166902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15772.260854] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [15772.283345] Lustre: 559093:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [15772.289060] Lustre: 559093:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [15772.512384] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15775.497751] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15781.639720] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15786.168678] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15792.432223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15792.495101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15792.670903] Lustre: lustre-OST0000: new disk, initializing [15792.672406] Lustre: Skipped 1 previous similar message [15792.675393] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15792.677153] Lustre: Skipped 2 previous similar messages [15794.641377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15794.644551] Lustre: Skipped 1 previous similar message [15794.647240] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15794.720510] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15796.353636] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15802.169846] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15805.042479] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15805.260380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15808.530563] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15808.745736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15815.146120] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15815.148698] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15815.150500] LustreError: Skipped 1 previous similar message [15815.161108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15815.166124] Lustre: Skipped 5 previous similar messages [15816.359697] Lustre: server umount lustre-OST0000 complete [15833.056193] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15833.184584] Lustre: server umount lustre-MDT0000 complete [15835.619298] LustreError: 559113:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15835.625201] LustreError: 559113:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 11 previous similar messages [15835.712618] LustreError: 558325:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743511691 with bad export cookie 4201005048283294420 [15835.715382] LustreError: 558325:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [15835.716600] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15835.891630] Lustre: server umount lustre-MDT0001 complete [15839.349317] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 08:48:14 (1743511694) [15851.107982] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [15863.563702] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [15873.716797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15878.537228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15883.158563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15889.143932] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15893.503685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15893.555491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15893.715944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [15893.737726] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15893.778284] Lustre: lustre-MDT0000: new disk, initializing [15893.828452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15893.830452] Lustre: Skipped 2 previous similar messages [15893.841402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15896.056830] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15903.307092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15903.382364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15903.472635] Lustre: 565264:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [15903.478402] Lustre: 565264:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [15903.655433] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15906.147793] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15911.425645] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15914.428715] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15918.695881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15918.787294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15920.506304] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15922.481420] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15927.647631] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15930.647792] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15930.829862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15933.089441] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15933.273903] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15935.971757] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15935.972964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15935.973978] LustreError: Skipped 1 previous similar message [15935.974759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15935.976818] Lustre: Skipped 4 previous similar messages [15935.980788] Lustre: Skipped 7 previous similar messages [15940.480508] Lustre: server umount lustre-OST0000 complete [15944.673376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15948.724208] Lustre: server umount lustre-MDT0000 complete [15950.912777] LustreError: 564494:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743511806 with bad export cookie 4201005048283295302 [15950.918313] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15951.111217] Lustre: server umount lustre-MDT0001 complete [15956.013703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15956.282246] LustreError: 568291:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15958.507879] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15963.423624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15965.915063] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15969.705060] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15972.758835] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15976.899322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15980.480313] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [15984.212622] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15993.831977] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15994.973932] Lustre: server umount lustre-OST0000 complete [16003.530746] Lustre: server umount lustre-MDT0000 complete [16005.757585] LustreError: 570816:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743511861 with bad export cookie 4201005048283296184 [16005.762811] LustreError: 570816:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [16005.765618] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16005.966198] Lustre: server umount lustre-MDT0001 complete [16121.757111] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [16124.605410] Key type lgssc unregistered [16124.978980] LNet: 571578:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16125.990601] LNet: Removed LNI 192.168.206.107@tcp [16126.818762] Key type .llcrypt unregistered [16126.820167] Key type ._llcrypt unregistered [16148.845338] Key type ._llcrypt registered [16148.846327] Key type .llcrypt registered [16148.942819] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [16168.582400] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [16171.038366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16171.059888] alg: No test for adler32 (adler32-zlib) [16172.282788] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [16172.597737] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [16172.599360] LNet: Accept secure, port 988 [16174.232242] Key type lgssc registered [16175.090300] Lustre: Echo OBD driver; http://www.lustre.org/ [16182.079427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16187.708623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16192.958492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16198.770703] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16203.378682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16203.432241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16203.441275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16204.620196] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [16204.644298] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16204.684403] Lustre: lustre-MDT0000: new disk, initializing [16204.751702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16204.766302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16207.227442] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [16214.191650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16214.263588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [16214.333568] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [16214.351838] Lustre: 575658:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [16214.373930] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16214.375419] Lustre: Skipped 1 previous similar message [16214.414147] Lustre: lustre-MDT0001: new disk, initializing [16214.490585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16214.515584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16214.519584] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16216.761352] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [16222.432492] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16226.408434] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16231.108090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16231.182153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [16231.338287] Lustre: 576607:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [16231.374130] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [16231.375173] Lustre: lustre-OST0000: new disk, initializing [16231.380534] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16231.467950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16234.888808] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [16236.530100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16236.533533] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16236.571529] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16240.049660] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [16242.596669] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16242.765837] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16244.986732] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16245.164268] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16246.758501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16246.758922] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16246.761242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16246.763407] Lustre: Skipped 1 previous similar message [16246.765895] Lustre: Skipped 1 previous similar message [16251.875582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16251.877395] Lustre: Skipped 1 previous similar message [16252.284284] Lustre: server umount lustre-OST0000 complete [16255.457081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16255.458968] LustreError: Skipped 1 previous similar message [16255.460073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16255.463532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16256.994354] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16260.463046] Lustre: server umount lustre-MDT0000 complete [16262.428199] LustreError: 578012:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743512117 with bad export cookie 10508993081862877905 [16262.431544] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16262.434056] LustreError: 578012:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [16262.604400] Lustre: server umount lustre-MDT0001 complete [16280.690755] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [16290.357596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16290.614577] LustreError: 579400:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16290.619569] LustreError: 579400:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [16290.647412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16293.024785] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [16295.906863] LustreError: 579400:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16298.204952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [16298.255369] Lustre: 580096:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [16298.395533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16300.764256] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [16302.600957] Lustre: Modifying parameter general.debug_raw_pointers in log params [16306.700982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [16306.812426] Lustre: 580835:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [16306.909027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16310.360505] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [16315.170369] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16315.238388] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [16315.327490] Lustre: 581634:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [16315.344649] Lustre: lustre-OST0001: new disk, initializing [16315.348068] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [16318.577878] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [16323.098338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [16323.102975] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [16323.179462] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [16328.258431] Lustre: DEBUG MARKER: Using TIMEOUT=20 [16340.259318] Lustre: Setting parameter general.lod.*.mdt_hash in log params [16365.943540] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 08:57:00 (1743512220) [16370.145394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16370.147581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16370.151881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16370.154302] Lustre: Skipped 2 previous similar messages [16373.953868] Lustre: server umount lustre-MDT0000 complete [16376.081853] LustreError: 579381:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743512231 with bad export cookie 10508993081862878787 [16376.086097] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16376.263768] Lustre: server umount lustre-MDT0001 complete [16387.780085] Lustre: server umount lustre-OST0000 complete [16400.096481] Lustre: server umount lustre-OST0001 complete [16404.991704] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [16417.893003] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [16428.072291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16433.048599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16437.555949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16442.919595] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16447.506770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16447.580234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16447.766219] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [16447.789763] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16447.829411] Lustre: lustre-MDT0000: new disk, initializing [16447.889430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16447.892396] Lustre: Skipped 1 previous similar message [16447.903538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16450.339354] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [16456.873421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16456.959853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [16457.028353] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [16457.044472] Lustre: 586336:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [16457.067076] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16457.069451] Lustre: Skipped 1 previous similar message [16457.110848] Lustre: lustre-MDT0001: new disk, initializing [16457.207824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16457.213417] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16459.530603] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [16464.628589] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16467.616595] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16471.717407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16471.790920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [16473.868702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16473.874510] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16473.960303] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16475.202951] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [16479.965348] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [16482.359452] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16482.510529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16484.713796] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16484.889159] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16489.445651] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16489.446574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16489.448240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16489.448247] Lustre: Skipped 4 previous similar messages [16489.448478] LustreError: Skipped 1 previous similar message [16489.451324] Lustre: Skipped 4 previous similar messages [16500.192147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16500.285186] Lustre: server umount lustre-OST0000 complete [16503.265059] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16503.266969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16508.538433] Lustre: server umount lustre-MDT0000 complete [16509.922974] LustreError: 586357:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16509.927227] LustreError: 586357:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [16510.553532] LustreError: 586351:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743512365 with bad export cookie 10508993081862880901 [16510.557070] LustreError: 586351:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 6 previous similar messages [16510.558557] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16510.731925] Lustre: server umount lustre-MDT0001 complete [16515.195082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16515.449848] LustreError: 589365:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16515.484847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16515.486541] Lustre: Skipped 2 previous similar messages [16517.522149] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [16520.674709] LustreError: 589365:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16522.053062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [16522.103511] Lustre: 590063:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [16522.106122] Lustre: 590063:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [16524.311473] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [16528.678241] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16532.828358] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16537.598211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [16541.118119] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [16545.152427] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [16693.728904] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16693.730707] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16693.733316] Lustre: Skipped 1 previous similar message [16693.734706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16693.736076] Lustre: Skipped 8 previous similar messages [16698.152523] Lustre: server umount lustre-OST0000 complete [16706.145489] Lustre: server umount lustre-MDT0000 complete [16708.044966] LustreError: 589346:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743512563 with bad export cookie 10508993081862881783 [16708.049598] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16708.221520] Lustre: server umount lustre-MDT0001 complete [16816.972182] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [16820.209784] Key type lgssc unregistered [16820.573653] LNet: 592605:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16821.606985] LNet: Removed LNI 192.168.206.107@tcp [16822.693143] Key type .llcrypt unregistered [16822.694231] Key type ._llcrypt unregistered [16849.641683] Key type ._llcrypt registered [16849.643085] Key type .llcrypt registered [16849.735309] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [16872.812039] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [16875.626711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16875.698654] alg: No test for adler32 (adler32-zlib) [16877.060092] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [16877.348932] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [16877.350939] LNet: Accept secure, port 988 [16879.000148] Key type lgssc registered [16880.158446] Lustre: Echo OBD driver; http://www.lustre.org/ [16889.340958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16896.032252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16902.199107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16908.730967] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16929.147822] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [16939.829813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16939.916178] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16939.931580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16941.131153] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [16941.152482] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16941.192671] Lustre: lustre-MDT0000: new disk, initializing [16941.249431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16941.261199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16944.006412] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [16952.351000] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16952.436159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [16952.503830] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [16952.520263] Lustre: 597402:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [16952.540638] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16952.542029] Lustre: Skipped 1 previous similar message [16952.584428] Lustre: lustre-MDT0001: new disk, initializing [16952.653123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16952.676537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16952.679399] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16955.600534] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [16958.851701] Lustre: Modifying parameter general.debug_raw_pointers in log params [16963.406170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16963.505377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [16963.670507] Lustre: 598208:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [16963.689287] Lustre: lustre-OST0000: new disk, initializing [16963.692679] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16963.805528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16967.003022] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [16969.205710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16969.210805] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16969.256829] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16973.148557] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16973.204855] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [16973.289255] Lustre: 599075:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [16973.305572] Lustre: lustre-OST0001: new disk, initializing [16973.308325] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [16973.360889] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [16976.168853] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [16982.035222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [16982.038460] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [16982.101134] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [16984.814520] Lustre: DEBUG MARKER: Using TIMEOUT=20 [16989.600911] Lustre: Setting parameter general.lod.*.mdt_hash in log params [17002.474068] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17002.477532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17002.480827] Lustre: Skipped 3 previous similar messages [17002.486392] Lustre: Skipped 3 previous similar messages [17007.587302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17007.590739] Lustre: Skipped 3 previous similar messages [17012.708562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17016.288132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [17016.455252] Lustre: server umount lustre-MDT0000 complete [17019.390997] LustreError: 596632:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743512874 with bad export cookie 11379546430681289466 [17019.399849] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17019.650532] Lustre: server umount lustre-MDT0001 complete [17033.263957] Lustre: server umount lustre-OST0000 complete [17046.144144] Lustre: server umount lustre-OST0001 complete [17055.773493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [17061.860645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [17062.386081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [17066.847987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [17073.194270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [17073.814948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [17092.569356] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [17102.516415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17102.523399] Lustre: lustre-MDT0000: reset Object Index mappings [17102.799319] LustreError: 604069:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17102.804965] LustreError: 604069:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 5 previous similar messages [17102.851429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17105.221562] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [17107.937238] LustreError: 604069:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17110.153204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [17110.160386] Lustre: lustre-MDT0001: reset Object Index mappings [17110.238359] Lustre: 604768:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [17112.518516] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [17114.174505] Lustre: Modifying parameter general.debug_raw_pointers in log params [17117.471700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [17117.574774] Lustre: 605518:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [17117.673608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17117.675173] Lustre: Skipped 1 previous similar message [17119.718467] LustreError: 605525:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17120.395185] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [17124.608245] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [17127.584797] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [17128.686928] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [17128.687018] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [17134.674095] Lustre: DEBUG MARKER: Using TIMEOUT=20 [17138.525310] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [17146.337084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17146.339052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17146.342506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17146.343939] Lustre: Skipped 3 previous similar messages [17149.412282] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17149.415226] Lustre: Skipped 2 previous similar messages [17152.118227] Lustre: server umount lustre-MDT0000 complete [17154.031437] LustreError: 604049:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743513009 with bad export cookie 11379546430681296256 [17154.034221] LustreError: 604049:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [17154.037239] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17154.212434] Lustre: server umount lustre-MDT0001 complete [17166.550337] Lustre: server umount lustre-OST0000 complete [17177.810810] Lustre: server umount lustre-OST0001 complete [17190.659165] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 09:10:45 (1743513045) [17199.119043] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [17207.691650] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [17214.944732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [17218.882125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [17222.890712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [17227.026940] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [17230.610076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [17230.662495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17230.811217] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [17230.830347] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17230.868673] Lustre: lustre-MDT0000: new disk, initializing [17230.915362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17230.917082] Lustre: Skipped 1 previous similar message [17230.928197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17232.827505] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [17239.412827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [17239.467032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [17239.541368] Lustre: 611981:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [17239.544099] Lustre: 611981:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [17239.562388] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17239.563880] Lustre: Skipped 1 previous similar message [17239.601686] Lustre: lustre-MDT0001: new disk, initializing [17239.677990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17239.680855] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17241.704390] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [17246.734636] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [17249.585328] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [17253.863181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [17253.920345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [17255.183917] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17255.277091] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17257.824608] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [17264.549749] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [17267.748905] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [17267.890727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [17270.076983] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [17270.217971] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [17275.874988] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17275.875642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [17275.876375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17275.876378] Lustre: Skipped 4 previous similar messages [17275.877636] Lustre: Skipped 1 previous similar message [17275.879285] LustreError: Skipped 1 previous similar message [17277.296313] Lustre: server umount lustre-OST0000 complete [17280.480995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17280.483068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17285.223651] Lustre: server umount lustre-MDT0000 complete [17286.113180] LustreError: 611233:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17286.117127] LustreError: 611233:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [17287.125426] LustreError: 611212:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743513142 with bad export cookie 11379546430681298349 [17287.128915] LustreError: 611212:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 5 previous similar messages [17287.129256] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17287.306686] Lustre: server umount lustre-MDT0001 complete [17304.616401] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [17311.681501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17311.880252] LustreError: 615721:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17311.905816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17311.907727] Lustre: Skipped 2 previous similar messages [17314.040643] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [17318.886731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [17318.932287] Lustre: 616418:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [17318.934745] Lustre: 616418:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [17321.348360] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [17323.270431] Lustre: Modifying parameter general.debug_raw_pointers in log params [17323.271952] Lustre: Skipped 1 previous similar message [17327.187411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [17330.715200] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [17335.589972] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [17335.659287] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [17335.765694] Lustre: lustre-OST0001: new disk, initializing [17335.766684] Lustre: Skipped 1 previous similar message [17335.769583] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [17335.770750] Lustre: Skipped 1 previous similar message [17337.816774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [17337.818907] Lustre: Skipped 1 previous similar message [17337.821083] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [17337.897051] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [17339.087589] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [17346.542541] Lustre: DEBUG MARKER: Using TIMEOUT=20 [17361.362918] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [17361.387564] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [17365.474059] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17365.475743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17365.478063] Lustre: Skipped 1 previous similar message [17365.479659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17365.481183] Lustre: Skipped 3 previous similar messages [17367.410336] Lustre: server umount lustre-MDT0000 complete [17368.548622] LustreError: 616439:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17368.551945] LustreError: 616439:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [17369.303659] LustreError: 615702:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743513224 with bad export cookie 11379546430681299231 [17369.308561] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17371.495257] Lustre: server umount lustre-MDT0001 complete [17382.921602] Lustre: server umount lustre-OST0000 complete [17395.311509] Lustre: server umount lustre-OST0001 complete [17418.902444] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [17429.198142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17429.420949] LustreError: 620993:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17429.426975] LustreError: 620993:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [17429.464380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17429.466113] Lustre: Skipped 3 previous similar messages [17431.643249] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [17436.831165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [17436.890726] Lustre: 621693:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.107@tcp: no remote llog for lustre-sptlrpc, check MGS config [17436.893795] Lustre: 621693:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 2 previous similar messages [17439.286433] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [17441.225712] Lustre: Modifying parameter general.debug_raw_pointers in log params [17441.227807] Lustre: Skipped 1 previous similar message [17445.743579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [17449.656942] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [17454.087264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [17456.174911] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [17457.556464] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [17460.655651] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [17467.889342] Lustre: DEBUG MARKER: Using TIMEOUT=20 [17481.849406] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [17481.887986] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [17494.648410] Lustre: DEBUG MARKER: == conf-sanity test 200a: check CPU partitions =========== 09:15:49 (1743513349) [17498.593903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17498.597222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17498.601420] Lustre: Skipped 3 previous similar messages [17498.603918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17500.223289] Lustre: server umount lustre-MDT0000 complete [17502.691107] LustreError: 621702:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17502.696885] LustreError: 621702:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 5 previous similar messages [17503.030852] LustreError: 620974:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743513358 with bad export cookie 11379546430681305524 [17503.035036] LustreError: 620974:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [17503.036367] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17503.231487] Lustre: server umount lustre-MDT0001 complete [17516.072854] Lustre: server umount lustre-OST0000 complete [17529.070064] Lustre: server umount lustre-OST0001 complete [17579.567268] Lustre: DEBUG MARKER: == conf-sanity test 200b: set CPU pattern using core selection ========================================================== 09:17:14 (1743513434) [17627.157533] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [17630.295554] Key type lgssc unregistered [17630.595439] LNet: 627690:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17631.654492] LNet: Removed LNI 192.168.206.107@tcp [17632.383106] Key type .llcrypt unregistered [17632.384266] Key type ._llcrypt unregistered [17720.908015] Key type ._llcrypt registered [17720.908939] Key type .llcrypt registered [17720.975745] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [17721.197669] Key type .llcrypt unregistered [17721.198573] Key type ._llcrypt unregistered [17756.517316] Key type ._llcrypt registered [17756.518317] Key type .llcrypt registered [17756.588517] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [17756.820652] Key type .llcrypt unregistered [17756.821580] Key type ._llcrypt unregistered [17844.406273] Key type ._llcrypt registered [17844.407140] Key type .llcrypt registered [17844.466185] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [17844.657325] Key type .llcrypt unregistered [17844.658149] Key type ._llcrypt unregistered [17872.975567] Key type ._llcrypt registered [17872.976387] Key type .llcrypt registered [17873.061477] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [17873.312675] Key type .llcrypt unregistered [17873.313594] Key type ._llcrypt unregistered [17901.048438] Key type ._llcrypt registered [17901.049603] Key type .llcrypt registered [17901.116148] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [17901.325635] Key type .llcrypt unregistered [17901.326576] Key type ._llcrypt unregistered [17986.971568] Key type ._llcrypt registered [17986.973078] Key type .llcrypt registered [17987.070699] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [17987.347707] Key type .llcrypt unregistered [17987.348703] Key type ._llcrypt unregistered [18025.495403] Key type ._llcrypt registered [18025.496429] Key type .llcrypt registered [18025.586911] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [18025.858678] Key type .llcrypt unregistered [18025.859744] Key type ._llcrypt unregistered [18057.420815] Key type ._llcrypt registered [18057.421792] Key type .llcrypt registered [18057.504346] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [18057.745748] Key type .llcrypt unregistered [18057.746642] Key type ._llcrypt unregistered [18119.863361] Key type ._llcrypt registered [18119.864393] Key type .llcrypt registered [18119.948440] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [18140.768659] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [18143.205416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [18143.228459] alg: No test for adler32 (adler32-zlib) [18144.517374] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [18144.855681] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [18144.857402] LNet: Accept secure, port 988 [18146.488205] Key type lgssc registered [18147.372272] Lustre: Echo OBD driver; http://www.lustre.org/ [18155.886078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [18162.953800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [18168.431361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [18174.383481] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [18176.392822] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 17589 sec ========= 09:27:11 (1743514031) [18178.911891] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 09:27:13 (1743514033) === [18184.401975] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 09:27:18 (1743514038) === [18216.203297] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [18218.629826] Key type lgssc unregistered [18218.890379] LNet: 644003:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [18218.897417] LNet: Removed LNI 192.168.206.107@tcp [18219.522859] Key type .llcrypt unregistered [18219.523872] Key type ._llcrypt unregistered