[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 628751692 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002174] x2apic enabled [ 0.003014] Switched APIC routing to physical x2apic. [ 0.006012] kvm-guest: setup PV IPIs [ 0.009380] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011011] pid_max: default: 32768 minimum: 301 [ 0.012137] LSM: Security Framework initializing [ 0.013059] Yama: becoming mindful. [ 0.014036] SELinux: Initializing. [ 0.015078] *** VALIDATE selinux *** [ 0.021612] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024157] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025128] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026160] *** VALIDATE tmpfs *** [ 0.027579] *** VALIDATE proc *** [ 0.028300] *** VALIDATE cgroup *** [ 0.029019] *** VALIDATE cgroup2 *** [ 0.030034] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031190] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032023] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034044] Spectre V2 : User space: Vulnerable [ 0.035020] Speculative Store Bypass: Vulnerable [ 0.038914] debug: unmapping init [mem 0xffffffffa4859000-0xffffffffa4860fff] [ 0.041221] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042892] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043034] ... version: 2 [ 0.044019] ... bit width: 48 [ 0.045022] ... generic registers: 4 [ 0.046020] ... value mask: 0000ffffffffffff [ 0.047017] ... max period: 00007fffffffffff [ 0.048016] ... fixed-purpose events: 3 [ 0.049013] ... event mask: 000000070000000f [ 0.050468] rcu: Hierarchical SRCU implementation. [ 0.052798] smp: Bringing up secondary CPUs ... [ 0.053913] x86: Booting SMP configuration: [ 0.054050] .... node #0, CPUs: #1 #2 #3 [ 0.062562] smp: Brought up 1 node, 4 CPUs [ 0.064015] smpboot: Max logical packages: 1 [ 0.065014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.109046] node 0 deferred pages initialised in 40ms [ 0.114827] devtmpfs: initialized [ 0.115351] x86/mm: Memory block size: 128MB [ 0.118225] gcov: version magic: 0x41383552 [ 0.121526] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.122108] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.123333] pinctrl core: initialized pinctrl subsystem [ 0.124256] [ 0.125008] ************************************************************* [ 0.126016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.127018] ** ** [ 0.128019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.129020] ** ** [ 0.130023] ** This means that this kernel is built to expose internal ** [ 0.131017] ** IOMMU data structures, which may compromise security on ** [ 0.132020] ** your system. ** [ 0.133015] ** ** [ 0.134026] ** If you see this message and you are not debugging the ** [ 0.135017] ** kernel, report this immediately to your vendor! ** [ 0.136016] ** ** [ 0.137014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.138014] ************************************************************* [ 0.140021] NET: Registered protocol family 16 [ 0.141508] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.142083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.143082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.144847] cpuidle: using governor menu [ 0.145000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.147674] PCI: Using configuration type 1 for base access [ 0.149246] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.162064] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.163020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.166805] cryptd: max_cpu_qlen set to 1000 [ 0.169682] ACPI: Added _OSI(Module Device) [ 0.170023] ACPI: Added _OSI(Processor Device) [ 0.171020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.172016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.176140] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.181134] ACPI: Interpreter enabled [ 0.182097] ACPI: PM: (supports S0 S3 S4 S5) [ 0.183014] ACPI: Using IOAPIC for interrupt routing [ 0.184234] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.185428] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.194258] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.195058] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.196032] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.197116] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.199415] acpiphp: Slot [2] registered [ 0.200136] acpiphp: Slot [3] registered [ 0.201107] acpiphp: Slot [4] registered [ 0.202121] acpiphp: Slot [5] registered [ 0.203148] acpiphp: Slot [6] registered [ 0.204132] acpiphp: Slot [7] registered [ 0.205172] acpiphp: Slot [8] registered [ 0.206142] acpiphp: Slot [9] registered [ 0.207129] acpiphp: Slot [10] registered [ 0.208245] acpiphp: Slot [11] registered [ 0.209161] acpiphp: Slot [12] registered [ 0.210177] acpiphp: Slot [13] registered [ 0.211127] acpiphp: Slot [14] registered [ 0.212122] acpiphp: Slot [15] registered [ 0.213122] acpiphp: Slot [16] registered [ 0.214117] acpiphp: Slot [17] registered [ 0.215150] acpiphp: Slot [18] registered [ 0.216122] acpiphp: Slot [19] registered [ 0.217110] acpiphp: Slot [20] registered [ 0.218119] acpiphp: Slot [21] registered [ 0.219097] acpiphp: Slot [22] registered [ 0.220100] acpiphp: Slot [23] registered [ 0.221152] acpiphp: Slot [24] registered [ 0.222129] acpiphp: Slot [25] registered [ 0.223203] acpiphp: Slot [26] registered [ 0.224113] acpiphp: Slot [27] registered [ 0.225320] acpiphp: Slot [28] registered [ 0.226135] acpiphp: Slot [29] registered [ 0.227118] acpiphp: Slot [30] registered [ 0.228131] acpiphp: Slot [31] registered [ 0.229054] PCI host bridge to bus 0000:00 [ 0.230028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.231028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.232044] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.233036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.234039] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.235033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.236203] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.239629] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.242631] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.249017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.252063] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.253034] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.254025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.255033] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.256647] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.257855] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.258047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.259694] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.261877] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.267840] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.270020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.274530] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.277027] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.280035] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.287027] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.291000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.300034] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.308027] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.325035] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.345343] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.358029] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.370031] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.389000] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.405216] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.428028] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.433043] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.444023] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.452264] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.457014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.462016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.477049] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.486658] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.494024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.499022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.511029] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.519146] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.520448] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.522410] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.524524] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.526349] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.530075] iommu: Default domain type: Passthrough [ 0.532655] SCSI subsystem initialized [ 0.534227] ACPI: bus type USB registered [ 0.535170] usbcore: registered new interface driver usbfs [ 0.537138] usbcore: registered new interface driver hub [ 0.539125] usbcore: registered new device driver usb [ 0.540207] pps_core: LinuxPPS API ver. 1 registered [ 0.542011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.545080] PTP clock support registered [ 0.547584] EDAC MC: Ver: 3.0.0 [ 0.549208] PCI: Using ACPI for IRQ routing [ 0.551042] NetLabel: Initializing [ 0.553016] NetLabel: domain hash size = 128 [ 0.554010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.555081] NetLabel: unlabeled traffic allowed by default [ 0.557297] vgaarb: loaded [ 0.559510] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.561015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.568000] clocksource: Switched to clocksource kvm-clock [ 0.699047] VFS: Disk quotas dquot_6.6.0 [ 0.703000] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.707487] *** VALIDATE ramfs *** [ 0.709266] *** VALIDATE hugetlbfs *** [ 0.710732] pnp: PnP ACPI init [ 0.713981] pnp: PnP ACPI: found 6 devices [ 0.744315] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.749299] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.751300] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.752669] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.755150] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.757867] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.760719] NET: Registered protocol family 2 [ 0.763427] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.769220] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.773337] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.779137] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.783285] TCP: Hash tables configured (established 65536 bind 65536) [ 0.786726] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.790365] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.794672] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.801604] NET: Registered protocol family 1 [ 0.809381] RPC: Registered named UNIX socket transport module. [ 0.813194] RPC: Registered udp transport module. [ 0.815919] RPC: Registered tcp transport module. [ 0.817811] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.821718] NET: Registered protocol family 44 [ 0.823427] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.826414] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.830217] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.833271] PCI: CLS 0 bytes, default 64 [ 0.837139] Unpacking initramfs... [ 3.159669] debug: unmapping init [mem 0xffff9c4dfcc54000-0xffff9c4dfffbffff] [ 3.170719] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.173948] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.178382] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.869481] Initialise system trusted keyrings [ 3.871110] Key type blacklist registered [ 3.873219] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.882466] zbud: loaded [ 3.885852] *** VALIDATE nfs *** [ 3.887178] *** VALIDATE nfs4 *** [ 3.888376] pstore: using deflate compression [ 3.892535] Platform Keyring initialized [ 4.044556] NET: Registered protocol family 38 [ 4.046275] Key type asymmetric registered [ 4.047842] Asymmetric key parser 'x509' registered [ 4.049822] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.053667] io scheduler mq-deadline registered [ 4.057187] io scheduler kyber registered [ 4.058781] io scheduler bfq registered [ 4.112118] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.114727] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.117665] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.120216] ACPI: Power Button [PWRF] [ 4.227639] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.343811] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.680758] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.823655] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.100247] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.145748] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.192055] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.202333] Non-volatile memory driver v1.3 [ 5.203767] Linux agpgart interface v0.103 [ 5.251667] virtio_blk virtio1: [vda] 133056 512-byte logical blocks (68.1 MB/65.0 MiB) [ 5.256255] vda: detected capacity change from 0 to 68124672 [ 5.287415] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.292637] vdb: detected capacity change from 0 to 1073741824 [ 5.310073] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.312984] vdc: detected capacity change from 0 to 2621440000 [ 5.329687] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.333289] vdd: detected capacity change from 0 to 2621440000 [ 5.358713] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.364035] vde: detected capacity change from 0 to 4294967296 [ 5.387766] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.390660] vdf: detected capacity change from 0 to 4294967296 [ 5.399652] libphy: Fixed MDIO Bus: probed [ 5.409963] usbcore: registered new interface driver usbserial_generic [ 5.411977] usbserial: USB Serial support registered for generic [ 5.414528] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.419702] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.421633] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.424403] mousedev: PS/2 mouse device common for all mice [ 5.428262] rtc_cmos 00:05: RTC can wake from S4 [ 5.431332] rtc_cmos 00:05: registered as rtc0 [ 5.433160] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.438543] intel_pstate: CPU model not supported [ 5.440766] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.444787] hid: raw HID events driver (C) Jiri Kosina [ 5.444990] usbcore: registered new interface driver usbhid [ 5.444993] usbhid: USB HID core driver [ 5.445160] drop_monitor: Initializing network drop monitor service [ 5.445296] Initializing XFRM netlink socket [ 5.445672] NET: Registered protocol family 10 [ 5.451522] Segment Routing with IPv6 [ 5.463305] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.464502] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.482244] NET: Registered protocol family 17 [ 5.485025] mpls_gso: MPLS GSO support [ 5.489633] RAS: Correctable Errors collector initialized. [ 5.491633] AVX version of gcm_enc/dec engaged. [ 5.493310] AES CTR mode by8 optimization enabled [ 5.614548] sched_clock: Marking stable (5614530247, 0)->(7007157749, -1392627502) [ 5.616971] registered taskstats version 1 [ 5.620753] Loading compiled-in X.509 certificates [ 5.622474] zswap: loaded using pool lzo/zbud [ 5.660399] Key type big_key registered [ 5.677363] Key type encrypted registered [ 5.679068] ima: No TPM chip found, activating TPM-bypass! [ 5.681668] ima: Allocated hash algorithm: sha1 [ 5.683249] ima: No architecture policies found [ 5.684503] evm: Initialising EVM extended attributes: [ 5.686234] evm: security.selinux [ 5.687247] evm: security.ima [ 5.688495] evm: security.capability [ 5.689990] evm: HMAC attrs: 0x1 [ 5.693963] rtc_cmos 00:05: setting system clock to 2025-09-02 10:33:52 UTC (1756809232) [ 5.703484] debug: unmapping init [mem 0xffffffffa5803000-0xffffffffa59fffff] [ 5.706793] debug: unmapping init [mem 0xffffffffa4582000-0xffffffffa4858fff] [ 5.718128] Write protecting the kernel read-only data: 28672k [ 5.720836] debug: unmapping init [mem 0xffffffffa2c03000-0xffffffffa2dfffff] [ 5.725021] debug: unmapping init [mem 0xffffffffa3514000-0xffffffffa35fffff] [ 5.770988] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.780025] systemd[1]: Detected virtualization kvm. [ 5.781361] systemd[1]: Detected architecture x86-64. [ 5.782592] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.819905] systemd[1]: No hostname configured. [ 5.821440] systemd[1]: Set hostname to . [ 5.822951] random: systemd: uninitialized urandom read (16 bytes read) [ 5.826360] systemd[1]: Initializing machine ID from random generator. [ 5.936675] random: ln: uninitialized urandom read (6 bytes read) [ 6.096744] random: systemd: uninitialized urandom read (16 bytes read) [ 6.098912] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 6.105261] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 6.107901] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.514499] device-mapper: uevent: version 1.0.3 [ 7.516830] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.425746] virtio_net virtio0 ens2: renamed from eth0 [ 9.444323] random: fast init done [ 10.149590] scsi host0: ata_piix [ 10.187244] scsi host1: ata_piix [ 10.188904] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.198152] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.202825] random: crng init done [ 15.204313] random: 7 urandom warning(s) missed due to ratelimiting [ 16.776866] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 18.500581] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.677337] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.143264] SELinux: Disabled at runtime. [ 21.238663] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.252672] systemd[1]: Detected virtualization kvm. [ 21.256086] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.230928] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.239605] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.251465] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.258720] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.270195] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.286417] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.300798] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ 22.502159] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 23.901373] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.852459] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.890117] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.736207] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.147244] EDAC sbridge: Ver: 1.1.2 [ 29.457793] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 29.907625] NFS: Registering the id_resolver key type [ 29.910748] Key type id_resolver registered [ 29.912921] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg415-server login: [ 69.010436] libcfs: loading out-of-tree module taints kernel. [ 69.037616] Key type ._llcrypt registered [ 69.039209] Key type .llcrypt registered [ 69.115053] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 79.895717] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 80.902695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 80.909764] alg: No test for adler32 (adler32-zlib) [ 81.023965] hrtimer: interrupt took 3794280 ns [ 82.052805] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 82.569932] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 84.232485] Key type lgssc registered [ 85.142737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 95.849121] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 97.228249] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 103.161886] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 107.524439] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 112.588138] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 123.015692] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 130.694462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 130.727970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 130.745386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 131.897308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 131.920918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 131.974170] Lustre: lustre-MDT0000: new disk, initializing [ 132.023054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 132.035126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 134.128667] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 142.608606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 142.662847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 142.709732] Lustre: 6490:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 142.746732] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 142.750532] Lustre: Skipped 1 previous similar message [ 142.813943] Lustre: lustre-MDT0001: new disk, initializing [ 142.858188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 142.886737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 142.896749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 145.381505] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 149.124492] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 155.707740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 155.789295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 155.930293] Lustre: lustre-OST0000: new disk, initializing [ 155.936023] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 155.983226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 156.823090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 156.828981] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 156.871148] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 160.067442] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 170.063134] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 170.107889] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 170.186053] Lustre: lustre-OST0001: new disk, initializing [ 170.191254] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 170.251737] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 174.221804] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 176.640059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 176.651298] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 176.699722] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 182.942483] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 187.157735] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 195.787847] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing check_logdir /tmp/testlogs/ [ 198.227955] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing yml_node [ 201.243674] Lustre: DEBUG MARKER: Client: 2.16.58.1 [ 202.936605] Lustre: DEBUG MARKER: MDS: 2.16.58.1 [ 204.582849] Lustre: DEBUG MARKER: OSS: 2.16.58.1 [ 205.648590] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 2 06:37:11 EDT 2025 [ 217.451391] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 218.541679] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 232.929627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 232.929920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 232.937418] Lustre: Skipped 2 previous similar messages [ 232.945681] Lustre: Skipped 3 previous similar messages [ 238.053335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 238.056845] Lustre: Skipped 3 previous similar messages [ 243.172259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 243.175294] Lustre: Skipped 3 previous similar messages [ 246.241399] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 246.337474] LustreError: 12190:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 246.385725] Lustre: server umount lustre-MDT0000 complete [ 248.673065] LustreError: 6482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756809475 with bad export cookie 7282574462534595757 [ 248.676286] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 248.678206] LustreError: 6482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 253.408632] LustreError: 10285:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 253.409304] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 253.409980] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 253.418054] LustreError: 10285:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 6 previous similar messages [ 253.423302] Lustre: Skipped 2 previous similar messages [ 254.812225] LustreError: 12393:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 254.814867] LustreError: 12393:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 254.953672] Lustre: server umount lustre-MDT0001 complete [ 257.421253] LustreError: 12595:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 257.425412] LustreError: 12595:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 257.477242] Lustre: server umount lustre-OST0000 complete [ 259.996895] LustreError: 12795:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 260.000504] LustreError: 12795:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 260.143510] Lustre: server umount lustre-OST0001 complete [ 264.334363] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 269.366891] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 275.031490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 278.904144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 282.833668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 286.968270] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 291.472735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 291.529728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 291.706962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 291.733630] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 291.778828] Lustre: lustre-MDT0000: new disk, initializing [ 291.850662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 291.865368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 294.147754] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 300.621294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 300.668902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 300.716555] Lustre: 14976:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 300.764593] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 300.768136] Lustre: Skipped 1 previous similar message [ 300.812398] Lustre: lustre-MDT0001: new disk, initializing [ 300.879229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 300.907411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 300.911138] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 303.021821] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 307.340453] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 309.250606] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 313.053698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 313.105889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 314.395399] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 314.473037] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 316.266645] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 320.307268] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 322.331272] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 322.457154] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 324.524288] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 324.632688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 329.697408] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 329.699982] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 329.701014] LustreError: Skipped 1 previous similar message [ 329.701880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 329.701885] Lustre: Skipped 1 previous similar message [ 329.718106] Lustre: Skipped 1 previous similar message [ 331.635154] LustreError: 17910:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 331.642434] LustreError: 17910:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 331.719908] Lustre: server umount lustre-OST0000 complete [ 334.819078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 339.662703] LustreError: 18112:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 339.664833] LustreError: 18112:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 339.712356] Lustre: server umount lustre-MDT0000 complete [ 339.937591] LustreError: 14985:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 339.956353] LustreError: 14985:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 342.074925] LustreError: 15756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756809568 with bad export cookie 7282574462534597290 [ 342.079963] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 342.088783] LustreError: 15756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 342.303225] Lustre: server umount lustre-MDT0001 complete [ 345.497784] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 06:39:31 (1756809571) [ 349.508064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 349.753279] LustreError: 18841:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 349.798184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 349.801013] Lustre: Skipped 1 previous similar message [ 351.904713] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 355.299857] LustreError: 18841:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 356.769569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 359.049292] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 361.985223] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 363.624363] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 367.004412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 370.598186] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 373.339819] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 380.780884] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 384.997304] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 384.999713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 385.002746] Lustre: Skipped 2 previous similar messages [ 385.008029] Lustre: Skipped 5 previous similar messages [ 390.681198] LustreError: 21142:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 390.685172] LustreError: 21142:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 390.759886] Lustre: server umount lustre-MDT0000 complete [ 392.866394] LustreError: 19560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756809619 with bad export cookie 7282574462534598172 [ 392.870262] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 392.873743] LustreError: 19560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 393.179584] Lustre: server umount lustre-MDT0001 complete [ 406.113369] Lustre: server umount lustre-OST0000 complete [ 416.713495] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 422.026553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 422.317824] LustreError: 22870:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 422.362686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 422.372609] Lustre: Skipped 2 previous similar messages [ 424.515996] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 429.787758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 432.467625] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 434.162668] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 438.406286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 442.031657] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 447.305456] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 447.359861] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 447.451300] Lustre: lustre-OST0001: new disk, initializing [ 447.453305] Lustre: Skipped 1 previous similar message [ 447.456458] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 447.458886] Lustre: Skipped 1 previous similar message [ 448.686631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 448.696752] Lustre: Skipped 1 previous similar message [ 448.700142] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 448.756160] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 450.466308] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 457.218270] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 459.178232] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 471.009951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 471.014548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 471.019649] Lustre: Skipped 1 previous similar message [ 471.022419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 471.025674] Lustre: Skipped 3 previous similar messages [ 472.805949] LustreError: 26354:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 472.808775] LustreError: 26354:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 472.865311] Lustre: server umount lustre-MDT0000 complete [ 474.598722] LustreError: 22869:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 474.605414] LustreError: 22869:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 475.047565] LustreError: 22851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756809701 with bad export cookie 7282574462534599019 [ 475.049150] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 475.053338] LustreError: 22851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 477.358211] Lustre: server umount lustre-MDT0001 complete [ 489.996262] Lustre: server umount lustre-OST0000 complete [ 501.498731] Lustre: server umount lustre-OST0001 complete [ 511.086936] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 516.718521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 516.971773] LustreError: 28131:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 517.025953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 517.028675] Lustre: Skipped 3 previous similar messages [ 519.357812] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 524.541578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 527.096773] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 528.846775] Lustre: 29248:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 532.728207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 536.779969] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 541.845503] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 545.246244] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 550.475784] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 552.778160] Lustre: 31100:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 563.684772] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 563.686195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 563.690256] Lustre: Skipped 6 previous similar messages [ 563.694602] Lustre: Skipped 3 previous similar messages [ 567.160397] LustreError: 31302:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 567.165442] LustreError: 31302:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 567.219831] Lustre: server umount lustre-MDT0000 complete [ 568.801390] LustreError: 28130:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.807925] LustreError: 28130:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 7 previous similar messages [ 569.033744] LustreError: 28112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756809795 with bad export cookie 7282574462534600314 [ 569.037353] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 569.039711] LustreError: 28112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 569.222880] Lustre: server umount lustre-MDT0001 complete [ 581.156906] Lustre: server umount lustre-OST0000 complete [ 593.754822] Lustre: server umount lustre-OST0001 complete [ 597.876969] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 06:43:44 (1756809824) [ 611.556230] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 616.682685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 619.130265] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 623.301318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 625.424768] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 626.740440] Lustre: 34998:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 629.837740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 632.641521] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 634.151638] LustreError: 35356:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 634.158877] LustreError: 35356:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 636.391494] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 639.546670] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 648.831851] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 650.529636] Lustre: 36859:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 657.378819] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 657.380501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 657.383209] Lustre: Skipped 3 previous similar messages [ 657.387155] Lustre: Skipped 2 previous similar messages [ 663.579742] Lustre: server umount lustre-MDT0000 complete [ 665.186268] LustreError: 33863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756809891 with bad export cookie 7282574462534601420 [ 665.189247] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 665.190997] LustreError: 33863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 665.340372] Lustre: server umount lustre-MDT0001 complete [ 677.843273] Lustre: server umount lustre-OST0000 complete [ 690.128311] Lustre: server umount lustre-OST0001 complete [ 693.887964] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 06:45:19 (1756809919) [ 700.773039] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 705.247463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 705.503946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 705.507194] Lustre: Skipped 7 previous similar messages [ 707.385452] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 711.116228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 712.796703] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 713.936545] Lustre: 40100:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 716.797487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 719.868946] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 724.587292] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 727.584655] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 736.911275] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 743.787786] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 743.790786] Lustre: Skipped 1 previous similar message [ 746.467038] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 746.472364] Lustre: Skipped 3 previous similar messages [ 751.053806] LustreError: 42159:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 751.057577] LustreError: 42159:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 751.090342] Lustre: server umount lustre-MDT0000 complete [ 752.861400] LustreError: 38965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756809979 with bad export cookie 7282574462534602526 [ 752.865504] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 752.867433] LustreError: 38965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 759.074507] Lustre: server umount lustre-MDT0001 complete [ 762.769589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 778.208991] LustreError: 3633:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9c4d411d6a00 x1842148078334848/t0(0) o250->MGC192.168.204.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 778.371169] LustreError: 40467:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 778.379588] LustreError: 40467:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 12 previous similar messages [ 780.134873] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 783.689986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 785.579760] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 787.635473] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 788.007231] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 791.345815] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 792.608957] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 795.401523] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 797.156461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 797.159020] Lustre: Skipped 18 previous similar messages [ 803.175733] Lustre: server umount lustre-MDT0000 complete [ 804.718414] LustreError: 40102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756810031 with bad export cookie 7282574462534603702 [ 804.721448] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 804.722635] LustreError: 40102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 804.871763] Lustre: server umount lustre-MDT0001 complete [ 816.687771] Lustre: server umount lustre-OST0000 complete [ 829.457129] Lustre: server umount lustre-OST0001 complete [ 836.042704] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 06:47:41 (1756810061) [ 844.483419] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 849.401031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 851.510566] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 855.807243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 857.781686] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 859.148355] Lustre: 47442:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 859.152721] Lustre: 47442:0:(mgs_llog.c:1345:mgs_modify_param()) Skipped 1 previous similar message [ 862.263640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 865.058574] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 869.113936] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 872.041787] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 881.563498] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 888.963944] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 895.458864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 895.464834] Lustre: Skipped 7 previous similar messages [ 901.580867] Lustre: server umount lustre-MDT0000 complete [ 903.602251] LustreError: 46301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756810130 with bad export cookie 7282574462534604486 [ 903.606074] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 903.614464] LustreError: 46301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 903.817795] Lustre: server umount lustre-MDT0001 complete [ 915.355554] Lustre: server umount lustre-OST0000 complete [ 926.359823] Lustre: server umount lustre-OST0001 complete [ 931.099436] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 06:49:17 (1756810157) [ 932.415594] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 933.366355] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 06:49:19 (1756810159) [ 938.078941] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 942.019468] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 946.328778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 949.047028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 952.372205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 952.410318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 952.560403] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 952.603888] Lustre: lustre-MDT0000: new disk, initializing [ 952.665674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 954.502538] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 958.560434] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 962.271038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 962.308056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 962.501238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 962.504424] Lustre: Skipped 10 previous similar messages [ 964.327062] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 964.352347] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 965.491555] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 969.845424] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 971.150335] Lustre: DEBUG MARKER: create test files [ 990.178108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 992.037598] Lustre: server umount lustre-OST0000 complete [ 994.322493] Lustre: server umount lustre-MDT0000 complete [ 999.802699] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1001.236134] Key type lgssc unregistered [ 1001.391484] LNet: 54854:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1001.395525] LNetError: 54854:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1001.405684] LNet: Removed LNI 192.168.204.115@tcp [ 1001.867251] Key type .llcrypt unregistered [ 1001.868884] Key type ._llcrypt unregistered [ 1016.844441] Key type ._llcrypt registered [ 1016.846130] Key type .llcrypt registered [ 1016.904269] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1017.461895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1017.472410] alg: No test for adler32 (adler32-zlib) [ 1018.388959] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 1018.531242] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1020.144272] Key type lgssc registered [ 1020.882727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1025.241391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1025.260887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1026.530733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1028.470543] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1030.950994] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1034.412579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1034.633930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1036.715347] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1036.765817] Lustre: 57434:0:(osd_internal.h:1341:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1036.770300] Lustre: 57434:0:(osd_handler.c:2079:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1036.773996] Lustre: 57434:0:(osd_handler.c:2086:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1036.778229] Lustre: 57434:0:(osd_handler.c:2096:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 1036.781940] Lustre: 57434:0:(osd_handler.c:2103:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1036.785815] Lustre: 57434:0:(osd_handler.c:2110:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1038.090119] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1041.022854] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1044.684540] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1047.802194] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1052.130264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1052.130728] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1052.138498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1056.440156] LustreError: 58176:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1056.535100] Lustre: server umount lustre-OST0000 complete [ 1058.545595] LustreError: 58378:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1058.548732] LustreError: 58378:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1058.680448] Lustre: server umount lustre-MDT0000 complete [ 1063.554283] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1064.789755] Key type lgssc unregistered [ 1064.944950] LNet: 58910:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1064.948721] LNetError: 58910:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1064.963925] LNet: Removed LNI 192.168.204.115@tcp [ 1065.377156] Key type .llcrypt unregistered [ 1065.378199] Key type ._llcrypt unregistered [ 1078.625881] Key type ._llcrypt registered [ 1078.627544] Key type .llcrypt registered [ 1078.681635] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1079.294212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1079.310227] alg: No test for adler32 (adler32-zlib) [ 1080.198929] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 1080.309243] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1081.904296] Key type lgssc registered [ 1082.431625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1085.627233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1085.637717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1086.827759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1088.373358] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1090.388532] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1093.059282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1093.210309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1095.572459] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1096.294608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1096.330243] Lustre: 61592:0:(osd_internal.h:1341:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1096.334766] Lustre: 61592:0:(osd_handler.c:2079:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1096.338094] Lustre: 61592:0:(osd_handler.c:2086:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1096.340676] Lustre: 61592:0:(osd_handler.c:2096:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 1096.344755] Lustre: 61592:0:(osd_handler.c:2103:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1096.347995] Lustre: 61592:0:(osd_handler.c:2110:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1097.756746] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1100.922477] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1105.710832] LustreError: 62285:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1105.776529] Lustre: server umount lustre-OST0000 complete [ 1117.668288] LustreError: 62485:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1117.670858] LustreError: 62485:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1117.760198] Lustre: server umount lustre-MDT0000 complete [ 1121.770449] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1123.021953] Key type lgssc unregistered [ 1123.172682] LNet: 63018:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1123.175856] LNetError: 63018:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1123.186586] LNet: Removed LNI 192.168.204.115@tcp [ 1123.572601] Key type .llcrypt unregistered [ 1123.574335] Key type ._llcrypt unregistered [ 1131.305327] Key type ._llcrypt registered [ 1131.307109] Key type .llcrypt registered [ 1131.359814] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 1138.695814] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1139.227687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1139.240877] alg: No test for adler32 (adler32-zlib) [ 1140.123695] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 1140.230417] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1141.848273] Key type lgssc registered [ 1142.311913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1146.304134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1149.279848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1152.016213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1154.987904] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1158.093950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1158.124566] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1158.132828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1159.239406] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1159.257104] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1159.297145] Lustre: lustre-MDT0000: new disk, initializing [ 1159.339298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1159.348398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1161.058271] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1166.060516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1166.093789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1166.138410] Lustre: 66121:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1166.167499] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1166.169958] Lustre: Skipped 1 previous similar message [ 1166.208087] Lustre: lustre-MDT0001: new disk, initializing [ 1166.246829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1166.264160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1166.268095] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1167.774965] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1171.491149] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1173.297219] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1176.763531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1176.794499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1176.919108] Lustre: lustre-OST0000: new disk, initializing [ 1176.921190] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1176.958277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1179.525092] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1183.241835] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1184.977417] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1185.282283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1185.286457] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1185.331666] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1186.109542] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1187.780471] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1187.874073] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1190.370731] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1190.371247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1190.378510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1190.383036] Lustre: Skipped 1 previous similar message [ 1194.733092] LustreError: 69069:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1194.815683] Lustre: server umount lustre-OST0000 complete [ 1197.025152] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1197.028871] LustreError: Skipped 1 previous similar message [ 1197.030910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1197.037245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1197.039275] Lustre: Skipped 1 previous similar message [ 1200.610527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1200.610990] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1202.503489] LustreError: 69270:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1202.507627] LustreError: 69270:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1202.548928] Lustre: server umount lustre-MDT0000 complete [ 1204.214079] LustreError: 66901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756810431 with bad export cookie 12021005877870560292 [ 1204.218934] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1204.219250] LustreError: 66901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1204.290979] LustreError: 69472:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1204.293915] LustreError: 69472:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1204.393666] Lustre: server umount lustre-MDT0001 complete [ 1208.136607] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 06:53:54 (1756810434) [ 1212.324422] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1218.158323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1228.929271] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 1232.975496] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1237.328534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1240.500112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1243.229482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1246.002112] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1248.889291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1248.920940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1249.032571] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1249.047595] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1249.088251] Lustre: lustre-MDT0000: new disk, initializing [ 1249.127863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1249.137574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1250.709306] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1255.519871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1255.551482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1255.594545] Lustre: 73702:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1255.623551] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1255.626894] Lustre: Skipped 1 previous similar message [ 1255.667611] Lustre: lustre-MDT0001: new disk, initializing [ 1255.710419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1255.726800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1255.730895] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1257.372410] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1261.146054] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1262.727535] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1265.756653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1265.798409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1265.929882] Lustre: lustre-OST0000: new disk, initializing [ 1265.932792] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1265.976839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1268.454579] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1272.145308] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1274.094961] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1274.376694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1274.383263] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1274.435283] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1275.223561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1276.882307] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1276.980899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1279.459237] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1279.459332] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1279.462496] LustreError: Skipped 1 previous similar message [ 1279.463228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1279.466963] Lustre: Skipped 1 previous similar message [ 1279.468985] Lustre: Skipped 1 previous similar message [ 1283.824134] LustreError: 76647:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1283.828177] LustreError: 76647:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1283.903501] Lustre: server umount lustre-OST0000 complete [ 1286.625536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1286.629613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1286.635433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1291.658242] LustreError: 76849:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1291.662128] LustreError: 76849:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1291.712426] Lustre: server umount lustre-MDT0000 complete [ 1293.411642] LustreError: 74482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756810520 with bad export cookie 12021005877870561160 [ 1293.416066] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1293.416475] LustreError: 74482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1293.588844] Lustre: server umount lustre-MDT0001 complete [ 1297.402380] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 06:55:23 (1756810523) [ 1300.418963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1300.689784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1302.284320] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1305.793402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1307.678328] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1310.061276] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1311.614661] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1314.419725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1316.946478] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1319.160258] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1320.800665] Lustre: 80058:0:(genops.c:1791:obd_export_evict_by_uuid()) MGS: evicting c97686bf-0b1e-4875-a3ae-24dcaff3dfee at adminstrative request [ 1320.932765] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1320.937990] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 1331.173785] Lustre: MGS: Client c97686bf-0b1e-4875-a3ae-24dcaff3dfee (at 0@lo) reconnecting [ 1331.179893] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0xa6d33127e95b8aec to 0xa6d33127e95b8d77 [ 1331.187625] Lustre: MGC192.168.204.115@tcp: Connection restored to 0@lo (at 0@lo) [ 1354.019230] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1354.053699] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1354.129968] Lustre: lustre-OST0001: new disk, initializing [ 1354.132816] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1354.178736] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1354.182730] Lustre: Skipped 2 previous similar messages [ 1356.944862] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1360.802736] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1360.923351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1360.927524] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1360.968415] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1365.988615] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1365.989514] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1365.992051] LustreError: Skipped 1 previous similar message [ 1365.992793] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1365.997569] Lustre: Skipped 2 previous similar messages [ 1365.999672] Lustre: Skipped 2 previous similar messages [ 1368.174220] LustreError: 81193:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 1368.176157] LustreError: 81193:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1368.197367] Lustre: server umount lustre-OST0001 complete [ 1371.108640] LustreError: 79317:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1371.119123] LustreError: 79317:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 1375.877201] Lustre: server umount lustre-OST0000 complete [ 1384.031091] Lustre: server umount lustre-MDT0000 complete [ 1385.667258] LustreError: 79339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756810612 with bad export cookie 12021005877870562679 [ 1385.670284] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1385.671941] LustreError: 79339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1385.759098] LustreError: 81796:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 1385.762665] LustreError: 81796:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 1385.877496] Lustre: server umount lustre-MDT0001 complete [ 1389.864465] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 06:56:55 (1756810615) [ 1390.662067] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 1391.582697] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 06:56:57 (1756810617) [ 1392.372450] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 1393.204390] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 06:56:59 (1756810619) [ 1394.011466] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 1394.863425] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 06:57:00 (1756810620) [ 1395.684218] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 1398.227562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1398.688442] loop: module loaded [ 1398.693578] loop0: detected capacity change from 0 to 4294967296 [ 1398.720161] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 1399.399179] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 1400.773372] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 1402.687215] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 1406.832385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1408.636994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1410.516391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1412.481392] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1415.945194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1415.985911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1416.005867] LustreError: 84366:0:(obd_config.c:782:class_setup()) setup lustre-OST0000-osd failed (-28) [ 1416.011158] LustreError: 84366:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 1416.015547] LustreError: 84366:0:(tgt_mount.c:2428:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 1416.020911] LustreError: 84366:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 1425.337252] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 1430.133738] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1434.899437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1438.186695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1441.178977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1444.197535] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1447.546115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1447.578318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1447.699463] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1447.715637] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1447.757168] Lustre: lustre-MDT0000: new disk, initializing [ 1447.796632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1447.806403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1449.583265] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1455.266790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1455.298481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1455.345613] Lustre: 87206:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1455.476760] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1457.146721] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1461.004112] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1462.885156] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1466.457497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1466.513900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1469.396991] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1473.029479] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1473.082201] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1473.320387] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1475.132727] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1475.239191] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1477.045469] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1477.161798] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1478.117393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1478.118651] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1478.124102] Lustre: Skipped 4 previous similar messages [ 1478.130929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1478.134055] Lustre: Skipped 5 previous similar messages [ 1480.115600] LustreError: 90143:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1480.119751] LustreError: 90143:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 1480.199063] Lustre: server umount lustre-OST0000 complete [ 1488.010217] Lustre: server umount lustre-MDT0000 complete [ 1488.353565] LustreError: 87213:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1488.365957] LustreError: 87213:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 1490.038746] LustreError: 87198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756810716 with bad export cookie 12021005877870563281 [ 1490.042658] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1490.043945] LustreError: 87198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1490.253905] Lustre: server umount lustre-MDT0001 complete [ 1494.680722] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 06:58:40 (1756810720) [ 1497.926560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1498.148061] LustreError: 91216:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1500.000824] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1503.202052] LustreError: 91216:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1503.749047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1505.521726] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1507.755875] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1509.357294] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1512.682740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1515.633110] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1518.185464] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1519.953911] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1524.170561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 1525.759220] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1525.863297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1528.816505] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1528.848447] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1528.913207] Lustre: lustre-OST0001: new disk, initializing [ 1528.915137] Lustre: Skipped 2 previous similar messages [ 1528.917763] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1528.920558] Lustre: Skipped 3 previous similar messages [ 1531.333760] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1535.187762] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1537.069943] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1538.578770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1538.585831] Lustre: Skipped 2 previous similar messages [ 1538.589358] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1538.620562] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1539.240599] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 1540.984228] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1541.082727] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1551.788946] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1561.435337] Lustre: Failing over lustre-MDT0000 [ 1561.789560] LustreError: 96141:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1561.792514] LustreError: 96141:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 1561.828921] Lustre: server umount lustre-MDT0000 complete [ 1562.897759] LustreError: 91212:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1562.905103] LustreError: 91212:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 1564.130560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1564.136712] Lustre: Skipped 5 previous similar messages [ 1573.126569] LustreError: 91212:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1573.133810] LustreError: 91212:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 11 previous similar messages [ 1577.066163] LDISKFS-fs (dm-0): recovery complete [ 1577.068673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1577.121833] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1577.224819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-60 [ 1577.228356] Lustre: Skipped 6 previous similar messages [ 1577.258246] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1578.256711] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 1578.257403] Lustre: 91213:0:(ldlm_lib.c:2020:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1578.749970] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1582.561960] Lustre: 93434:0:(ldlm_lib.c:2020:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1582.567114] Lustre: 93434:0:(ldlm_lib.c:2020:extend_recovery_timer()) Skipped 1 previous similar message [ 1582.577564] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1582.603655] LustreError: 96888:0:(ldlm_lib.c:2632:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1582.920131] LustreError: 96888:0:(ldlm_lib.c:2632:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1583.242574] LustreError: 96888:0:(ldlm_lib.c:2632:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1583.247091] LustreError: 96888:0:(ldlm_lib.c:2632:replay_request_or_update()) Skipped 1 previous similar message [ 1583.560587] LustreError: 96888:0:(ldlm_lib.c:2632:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1583.565519] LustreError: 96888:0:(ldlm_lib.c:2632:replay_request_or_update()) Skipped 1 previous similar message [ 1583.571127] Lustre: 96888:0:(ldlm_lib.c:2020:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1583.576607] Lustre: 96888:0:(ldlm_lib.c:2020:extend_recovery_timer()) Skipped 7 previous similar messages [ 1584.514606] LustreError: 96888:0:(ldlm_lib.c:2632:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1584.517612] LustreError: 96888:0:(ldlm_lib.c:2632:replay_request_or_update()) Skipped 3 previous similar messages [ 1584.832180] LustreError: 96888:0:(ldlm_lib.c:2632:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1584.835620] LustreError: 96888:0:(ldlm_lib.c:2632:replay_request_or_update()) Skipped 3 previous similar messages [ 1585.778594] Lustre: 96888:0:(ldlm_lib.c:2020:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1585.783084] Lustre: 96888:0:(ldlm_lib.c:2020:extend_recovery_timer()) Skipped 6 previous similar messages [ 1586.721020] LustreError: 96888:0:(ldlm_lib.c:2632:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1586.725324] LustreError: 96888:0:(ldlm_lib.c:2632:replay_request_or_update()) Skipped 6 previous similar messages [ 1587.040133] LustreError: 96888:0:(ldlm_lib.c:2632:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1587.043033] LustreError: 96888:0:(ldlm_lib.c:2632:replay_request_or_update()) Skipped 6 previous similar messages [ 1589.857024] Lustre: 96888:0:(ldlm_lib.c:2020:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1589.862347] Lustre: 96888:0:(ldlm_lib.c:2020:extend_recovery_timer()) Skipped 12 previous similar messages [ 1590.803927] LustreError: 96888:0:(ldlm_lib.c:2632:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1590.808888] LustreError: 96888:0:(ldlm_lib.c:2632:replay_request_or_update()) Skipped 12 previous similar messages [ 1591.128130] LustreError: 96888:0:(ldlm_lib.c:2632:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1591.130743] LustreError: 96888:0:(ldlm_lib.c:2632:replay_request_or_update()) Skipped 12 previous similar messages [ 1598.001379] Lustre: 96888:0:(ldlm_lib.c:2020:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1598.005817] Lustre: 96888:0:(ldlm_lib.c:2020:extend_recovery_timer()) Skipped 25 previous similar messages [ 1598.947076] LustreError: 96888:0:(ldlm_lib.c:2632:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1598.951134] LustreError: 96888:0:(ldlm_lib.c:2632:replay_request_or_update()) Skipped 25 previous similar messages [ 1599.264131] LustreError: 96888:0:(ldlm_lib.c:2632:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1599.267615] LustreError: 96888:0:(ldlm_lib.c:2632:replay_request_or_update()) Skipped 25 previous similar messages [ 1614.248880] Lustre: 96888:0:(ldlm_lib.c:2020:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1614.253167] Lustre: 96888:0:(ldlm_lib.c:2020:extend_recovery_timer()) Skipped 51 previous similar messages [ 1615.194714] LustreError: 96888:0:(ldlm_lib.c:2632:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1615.198718] LustreError: 96888:0:(ldlm_lib.c:2632:replay_request_or_update()) Skipped 51 previous similar messages [ 1615.512138] LustreError: 96888:0:(ldlm_lib.c:2632:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1615.516512] LustreError: 96888:0:(ldlm_lib.c:2632:replay_request_or_update()) Skipped 51 previous similar messages [ 1638.296161] Lustre: 96888:0:(ldlm_lib.c:2328:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 1638.300244] Lustre: 96888:0:(ldlm_lib.c:2338:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1638.303645] Lustre: 96888:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client a1afdbb4-1a0c-451b-96fb-b8d0d1a27ff7@192.168.204.15@tcp [ 1638.308548] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1638.311332] Lustre: 96888:0:(ldlm_lib.c:1850:abort_req_replay_queue()) @@@ aborted: req@00000000886233d8 x1842149179067008/t0(8589934775) o101->a1afdbb4-1a0c-451b-96fb-b8d0d1a27ff7@192.168.204.15@tcp:620/0 lens 592/0 e 0 to 0 dl 1756810875 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 1638.321090] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1638.326156] Lustre: lustre-MDT0000: Denying connection for new client a1afdbb4-1a0c-451b-96fb-b8d0d1a27ff7 (at 192.168.204.15@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) already passed deadline 0:00 [ 1638.329531] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 1638.345620] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1638.345705] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 1638.348724] Lustre: Skipped 2 previous similar messages [ 1638.367455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:78 to 0x280000401:577) [ 1638.367493] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:47 to 0x2c0000401:545) [ 1644.004914] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1644.005267] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1644.009025] LustreError: Skipped 1 previous similar message [ 1644.015033] Lustre: Skipped 1 previous similar message [ 1644.016141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1644.023743] Lustre: Skipped 3 previous similar messages [ 1645.054373] Lustre: server umount lustre-OST0000 complete [ 1647.010477] LustreError: 92814:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1647.016414] LustreError: 92814:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 5 previous similar messages [ 1652.669370] Lustre: server umount lustre-OST0001 complete [ 1660.258342] Lustre: server umount lustre-MDT0000 complete [ 1661.724500] LustreError: 91196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756810888 with bad export cookie 12021005877870614479 [ 1661.729169] LustreError: 91196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1661.729512] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1661.903840] Lustre: server umount lustre-MDT0001 complete [ 1665.146233] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 07:01:31 (1756810891) [ 1667.879393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1667.882558] Lustre: *** cfs_fail_loc=197, val=0*** [ 1684.384583] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1690.528451] LustreError: 98858:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.115@tcp: failed processing log, type 4: rc = -110 [ 1696.681384] Lustre: *** cfs_fail_loc=197, val=0*** [ 1696.683017] Lustre: Skipped 65 previous similar messages [ 1721.279867] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1723.255621] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1724.018936] LustreError: 99523:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1724.022023] LustreError: 99523:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 1724.091524] Lustre: server umount lustre-OST0000 complete [ 1727.629746] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 07:02:33 (1756810953) [ 1741.957771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1742.187228] LustreError: 100368:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1742.196936] LustreError: 100368:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 1743.877615] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1747.161941] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1749.298283] LustreError: 100399:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1749.302251] LustreError: 100399:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1749.305842] LustreError: 100399:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 1749.492816] Lustre: server umount lustre-MDT0000 complete [ 1755.460498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1762.606493] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 1766.635343] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1770.759988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1773.368428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1775.651524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1778.009456] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1780.588516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1780.614345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1780.723849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1780.739508] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1780.782546] Lustre: lustre-MDT0000: new disk, initializing [ 1780.819943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1782.385379] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1787.031285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1787.054736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1787.090466] Lustre: 104431:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1787.190503] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1788.589419] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1791.962912] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1793.326413] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1795.786373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1795.809464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1797.981652] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1801.178832] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1802.290413] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1802.710638] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1802.797819] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1804.180958] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1804.262425] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1807.329684] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1807.330604] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1807.333456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1807.333463] Lustre: Skipped 6 previous similar messages [ 1807.333616] LustreError: Skipped 1 previous similar message [ 1807.341655] Lustre: Skipped 6 previous similar messages [ 1811.050784] Lustre: server umount lustre-OST0000 complete [ 1818.516810] Lustre: server umount lustre-MDT0000 complete [ 1819.742455] LustreError: 106177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756811046 with bad export cookie 12021005877870625084 [ 1819.745105] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1819.746117] LustreError: 106177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1819.890103] Lustre: server umount lustre-MDT0001 complete [ 1823.050674] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 07:04:09 (1756811049) [ 1826.553764] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1827.744252] Key type lgssc unregistered [ 1827.868350] LNet: 108499:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1827.871372] LNetError: 108499:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1827.885457] LNet: Removed LNI 192.168.204.115@tcp [ 1828.234887] Key type .llcrypt unregistered [ 1828.236076] Key type ._llcrypt unregistered [ 1834.842379] Key type ._llcrypt registered [ 1834.844435] Key type .llcrypt registered [ 1834.884860] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 1841.084174] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1841.491584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1841.500608] alg: No test for adler32 (adler32-zlib) [ 1842.359740] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 1842.447914] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1844.032159] Key type lgssc registered [ 1844.469925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1847.805254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1850.244702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1852.616323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1854.954427] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1857.128235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1859.682908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1859.707076] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1859.714396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1860.827359] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1860.842464] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1860.883193] Lustre: lustre-MDT0000: new disk, initializing [ 1860.922561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1860.934870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1862.561949] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1867.198380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1867.225596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1867.333517] Lustre: lustre-OST0000: new disk, initializing [ 1867.336244] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1867.338724] Lustre: Skipped 1 previous similar message [ 1867.371723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1869.586390] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1873.926390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1873.931266] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1873.943637] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1874.370933] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1874.399648] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1874.466889] Lustre: lustre-OST0001: new disk, initializing [ 1874.469964] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1874.504420] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1876.769650] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1884.641700] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1884.642389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1884.645553] Lustre: Skipped 1 previous similar message [ 1884.649715] Lustre: Skipped 1 previous similar message [ 1887.297346] LustreError: 114044:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1887.404853] Lustre: server umount lustre-MDT0000 complete [ 1890.058170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1892.237268] LustreError: 113457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756811119 with bad export cookie 3556303987245660108 [ 1892.239977] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1892.242478] LustreError: 113457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1892.265237] LustreError: 114641:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 1892.267298] LustreError: 114641:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1892.287612] Lustre: server umount lustre-OST0000 complete [ 1893.822749] LustreError: 114841:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1893.825573] LustreError: 114841:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1893.898117] Lustre: server umount lustre-OST0001 complete [ 1897.362171] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 07:05:23 (1756811123) [ 1901.377086] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1904.956952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1905.382563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1907.680804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1908.134336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1910.751703] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 07:05:36 (1756811136) [ 1915.734112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1916.438169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1917.155761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1917.987106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1918.405110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1919.240146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1925.760198] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 1929.552071] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1933.522094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1936.045068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1938.515503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1941.025411] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1943.766512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1943.799162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1943.913964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1943.933453] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1943.978313] Lustre: lustre-MDT0000: new disk, initializing [ 1944.010069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1944.018927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1945.508572] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1950.197390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1950.230610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1950.274340] Lustre: 120157:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1950.301293] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1950.303557] Lustre: Skipped 1 previous similar message [ 1950.339520] Lustre: lustre-MDT0001: new disk, initializing [ 1950.381183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1950.395963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1950.402146] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1951.841564] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1955.249171] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1956.766456] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1959.287439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1959.318423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1959.449481] Lustre: lustre-OST0000: new disk, initializing [ 1959.453248] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1959.499323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1961.664755] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1961.669421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1961.673846] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1961.715686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1964.911244] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1966.447367] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1966.537189] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1968.162064] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1968.274471] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1972.194754] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1972.197447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1972.198763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1972.199039] LustreError: Skipped 1 previous similar message [ 1972.207769] Lustre: Skipped 1 previous similar message [ 1975.152186] LustreError: 123097:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1975.155072] LustreError: 123097:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1975.241797] Lustre: server umount lustre-OST0000 complete [ 1977.314100] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1977.314980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1977.318874] Lustre: Skipped 1 previous similar message [ 1977.321353] Lustre: Skipped 1 previous similar message [ 1982.433410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1982.435705] Lustre: Skipped 1 previous similar message [ 1982.782539] LustreError: 123299:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1982.785669] LustreError: 123299:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1982.823723] Lustre: server umount lustre-MDT0000 complete [ 1984.044934] LustreError: 121905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756811210 with bad export cookie 3556303987245661256 [ 1984.047786] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1984.050357] LustreError: 121905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1984.175683] Lustre: server umount lustre-MDT0001 complete [ 1986.789280] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 07:06:53 (1756811213) [ 1988.964657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1989.146747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1990.372329] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1993.150525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1994.603729] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1996.344870] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1997.525567] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1999.652959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2001.580703] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2003.432174] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2021.253936] Lustre: *** cfs_fail_loc=159, val=0*** [ 2037.529199] Lustre: lustre-MDT0000: Client 99cf356a-5b1c-40c3-a35c-e0bba00d25d1 (at 192.168.204.15@tcp) reconnecting [ 2038.042703] Lustre: *** cfs_fail_loc=159, val=0*** [ 2038.044441] Lustre: Skipped 5 previous similar messages [ 2053.395196] Lustre: lustre-MDT0000: Client 99cf356a-5b1c-40c3-a35c-e0bba00d25d1 (at 192.168.204.15@tcp) reconnecting [ 2070.497105] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2070.499961] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2070.504216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2071.521545] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2075.309049] LustreError: 126683:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2075.311693] LustreError: 126683:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2075.366646] Lustre: server umount lustre-OST0000 complete [ 2076.641335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2076.646158] Lustre: Skipped 2 previous similar messages [ 2081.761171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2081.762890] Lustre: Skipped 5 previous similar messages [ 2082.711586] Lustre: server umount lustre-MDT0000 complete [ 2083.941071] LustreError: 126581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756811310 with bad export cookie 3556303987245662124 [ 2083.944735] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2083.945331] LustreError: 126581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2084.090436] Lustre: server umount lustre-MDT0001 complete [ 2087.969920] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2089.108303] Key type lgssc unregistered [ 2089.246430] LNet: 127618:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2089.249688] LNetError: 127618:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2089.259459] LNet: Removed LNI 192.168.204.115@tcp [ 2089.596601] Key type .llcrypt unregistered [ 2089.597966] Key type ._llcrypt unregistered [ 2098.844469] Key type ._llcrypt registered [ 2098.845431] Key type .llcrypt registered [ 2098.885614] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2099.329669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2099.351841] alg: No test for adler32 (adler32-zlib) [ 2100.223378] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 2100.313239] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2101.896147] Key type lgssc registered [ 2102.295475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2105.070923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2105.082529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2106.244176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2107.555465] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2110.375423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2110.507348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2111.879193] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2113.661140] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2114.874485] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2117.098731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2117.208413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2118.946192] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2120.626188] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2131.558030] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 2140.763838] Lustre: *** cfs_fail_loc=159, val=0*** [ 2156.306529] Lustre: lustre-MDT0000: Client 253e1fd3-ece8-4a90-b98c-3c221955f5f2 (at 192.168.204.15@tcp) reconnecting [ 2156.985774] Lustre: *** cfs_fail_loc=159, val=0*** [ 2173.205301] Lustre: lustre-MDT0001: Client 253e1fd3-ece8-4a90-b98c-3c221955f5f2 (at 192.168.204.15@tcp) reconnecting [ 2173.668305] Lustre: *** cfs_fail_loc=159, val=0*** [ 2173.669428] Lustre: Skipped 3 previous similar messages [ 2190.096927] Lustre: lustre-MDT0001: Client 253e1fd3-ece8-4a90-b98c-3c221955f5f2 (at 192.168.204.15@tcp) reconnecting [ 2191.243813] Lustre: 131717:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2191.965467] Lustre: *** cfs_fail_loc=159, val=0*** [ 2191.967390] Lustre: Skipped 4 previous similar messages [ 2207.510697] Lustre: lustre-MDT0000: Client 976666c0-3f2a-45b8-9410-64ee2cc5e6e4 (at 192.168.204.15@tcp) reconnecting [ 2208.150981] Lustre: *** cfs_fail_loc=159, val=0*** [ 2208.152814] Lustre: Skipped 14 previous similar messages [ 2223.376411] Lustre: lustre-MDT0000: Client 976666c0-3f2a-45b8-9410-64ee2cc5e6e4 (at 192.168.204.15@tcp) reconnecting [ 2223.838022] Lustre: 132005:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2243.553567] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2243.557423] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2243.563330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2244.578606] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2244.581253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2244.582813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2244.586727] Lustre: Skipped 1 previous similar message [ 2247.853463] LustreError: 132129:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2247.919400] Lustre: server umount lustre-OST0000 complete [ 2249.697276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2249.697881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2249.702199] Lustre: Skipped 1 previous similar message [ 2249.705474] Lustre: Skipped 1 previous similar message [ 2254.816931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2254.818944] Lustre: Skipped 1 previous similar message [ 2255.293243] LustreError: 132330:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2255.295121] LustreError: 132330:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2255.318974] Lustre: server umount lustre-MDT0000 complete [ 2256.513983] LustreError: 128972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756811483 with bad export cookie 2958416399880812234 [ 2256.516554] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2256.518483] LustreError: 128972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2256.580452] LustreError: 132531:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2256.582111] LustreError: 132531:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2256.656060] Lustre: server umount lustre-MDT0001 complete [ 2260.208832] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2261.296469] Key type lgssc unregistered [ 2261.433314] LNet: 133061:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2261.435758] LNetError: 133061:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2261.442232] LNet: Removed LNI 192.168.204.115@tcp [ 2261.775050] Key type .llcrypt unregistered [ 2261.776278] Key type ._llcrypt unregistered [ 2270.551124] Key type ._llcrypt registered [ 2270.552827] Key type .llcrypt registered [ 2270.592059] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2270.894632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2270.900067] alg: No test for adler32 (adler32-zlib) [ 2271.753429] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 2271.838265] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2273.416118] Key type lgssc registered [ 2273.797882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2276.382717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2276.391482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2277.545856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2278.852613] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2281.727061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2281.854652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2283.214578] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2284.958872] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2286.159420] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2288.529044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2288.640455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2290.639889] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2291.751631] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 2292.537785] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2303.017536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 2305.375036] Lustre: 136706:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2305.980341] Lustre: 136754:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2306.398420] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 2307.459495] Lustre: 136901:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 2307.752395] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 2307.754881] Lustre: Skipped 1 previous similar message [ 2308.043379] Lustre: 136999:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2312.160889] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2312.163798] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2312.167371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2313.186238] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2313.187724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2313.189298] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2314.989081] LustreError: 137096:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2315.065936] Lustre: server umount lustre-OST0000 complete [ 2317.792795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2317.795909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2317.801478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2317.803720] Lustre: Skipped 1 previous similar message [ 2322.430827] LustreError: 137305:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2322.432784] LustreError: 137305:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2322.461198] Lustre: server umount lustre-MDT0000 complete [ 2323.425112] LustreError: 134436:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2323.432035] LustreError: 134436:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 2323.851512] LustreError: 136055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756811550 with bad export cookie 15909453237751230437 [ 2323.855097] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2323.856342] LustreError: 136055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2323.926652] LustreError: 137507:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2323.929439] LustreError: 137507:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2324.024387] Lustre: server umount lustre-MDT0001 complete [ 2328.086471] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2329.296256] Key type lgssc unregistered [ 2329.446683] LNet: 138038:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2329.449673] LNetError: 138038:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2329.457524] LNet: Removed LNI 192.168.204.115@tcp [ 2329.814222] Key type .llcrypt unregistered [ 2329.815373] Key type ._llcrypt unregistered [ 2338.855869] Key type ._llcrypt registered [ 2338.857133] Key type .llcrypt registered [ 2338.895035] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2339.216409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2339.251027] alg: No test for adler32 (adler32-zlib) [ 2340.116174] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 2340.209643] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2341.800184] Key type lgssc registered [ 2342.186566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2344.829050] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2344.836645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2345.997110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2347.310726] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2350.232058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2350.369418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2351.757192] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2353.570698] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2354.786959] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2357.016659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2357.142681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2359.020525] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2360.234814] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 2360.825523] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2370.470886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 2373.388704] Lustre: *** cfs_fail_loc=159, val=0*** [ 2389.781869] Lustre: lustre-MDT0000: Client d1d834b4-b1ee-4140-b5d3-b5ed07309d18 (at 192.168.204.15@tcp) reconnecting [ 2391.013959] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2391.014108] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2391.020952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2391.023066] LustreError: Skipped 1 previous similar message [ 2391.025436] Lustre: Skipped 1 previous similar message [ 2396.130495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2396.133271] Lustre: Skipped 1 previous similar message [ 2397.101725] LustreError: 141807:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2397.166574] Lustre: server umount lustre-OST0000 complete [ 2401.249475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2401.250174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2401.252817] Lustre: Skipped 2 previous similar messages [ 2404.542411] LustreError: 142008:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2404.544552] LustreError: 142008:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2404.572760] Lustre: server umount lustre-MDT0000 complete [ 2405.818398] LustreError: 141036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756811632 with bad export cookie 3396900762568991506 [ 2405.821760] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2405.822867] LustreError: 141036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2405.890548] LustreError: 142210:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2405.892721] LustreError: 142210:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2405.968787] Lustre: server umount lustre-MDT0001 complete [ 2409.634542] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2410.804877] Key type lgssc unregistered [ 2410.940372] LNet: 142741:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2410.943522] LNetError: 142741:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2410.950291] LNet: Removed LNI 192.168.204.115@tcp [ 2411.281631] Key type .llcrypt unregistered [ 2411.282818] Key type ._llcrypt unregistered [ 2421.008886] Key type ._llcrypt registered [ 2421.010151] Key type .llcrypt registered [ 2421.047688] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2421.523425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2421.538884] alg: No test for adler32 (adler32-zlib) [ 2422.402717] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 2422.504115] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2424.080154] Key type lgssc registered [ 2424.436318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2426.976053] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2426.983044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2428.135376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2429.431609] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2432.389886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2432.517123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2433.878501] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2435.797367] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2437.048223] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2439.253219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2439.361456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2441.224840] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2442.474217] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 2443.122285] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2453.738331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 2466.546913] Lustre: lustre-OST0000: evicting 7238f19e-e048-41c8-a97c-b0191d831c19 (at 192.168.204.15@tcp) by administrative request [ 2466.549456] Lustre: Skipped 1 previous similar message [ 2493.921277] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2493.923808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2493.929515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2494.948253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2494.949542] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2494.950127] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2496.686114] LustreError: 146916:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2496.741492] Lustre: server umount lustre-OST0000 complete [ 2499.040793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2499.042513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2499.045766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2499.048124] Lustre: Skipped 1 previous similar message [ 2505.184797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2505.187249] Lustre: Skipped 3 previous similar messages [ 2510.304924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2510.306954] Lustre: Skipped 1 previous similar message [ 2512.352130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2512.385856] LustreError: 147117:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2512.387620] LustreError: 147117:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2512.412702] Lustre: server umount lustre-MDT0000 complete [ 2513.652987] LustreError: 144838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756811740 with bad export cookie 101783672254694877 [ 2513.655476] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2513.657170] LustreError: 144838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2513.729271] LustreError: 147321:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2513.730960] LustreError: 147321:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2513.824639] Lustre: server umount lustre-MDT0001 complete [ 2517.312205] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2518.375816] Key type lgssc unregistered [ 2518.516578] LNet: 147852:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2518.518646] LNetError: 147852:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2518.526378] LNet: Removed LNI 192.168.204.115@tcp [ 2518.876494] Key type .llcrypt unregistered [ 2518.878189] Key type ._llcrypt unregistered [ 2545.325443] Key type ._llcrypt registered [ 2545.326254] Key type .llcrypt registered [ 2545.362680] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2545.743289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2545.755967] alg: No test for adler32 (adler32-zlib) [ 2546.615163] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 2546.712312] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2548.304167] Key type lgssc registered [ 2548.722931] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2551.391948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2551.399766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2552.572092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2553.972910] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2556.926181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2557.068639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2558.496800] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2560.324937] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2561.601308] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2563.947428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2564.069913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2566.000319] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2567.144703] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 2567.863498] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2577.383836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2582.498464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2582.500050] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2582.501381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2582.503561] Lustre: Skipped 1 previous similar message [ 2586.926227] LustreError: 152584:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2586.991689] Lustre: server umount lustre-OST0000 complete [ 2592.737393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2592.738293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2592.740529] Lustre: Skipped 1 previous similar message [ 2592.742867] Lustre: Skipped 1 previous similar message [ 2594.490127] LustreError: 152791:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2594.491931] LustreError: 152791:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2594.514923] Lustre: server umount lustre-MDT0000 complete [ 2595.661383] LustreError: 150288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756811822 with bad export cookie 13077315300045946471 [ 2595.663841] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2595.666635] LustreError: 150288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2595.721160] LustreError: 152995:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2595.723705] LustreError: 152995:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2595.799427] Lustre: server umount lustre-MDT0001 complete [ 2599.390229] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2600.445276] Key type lgssc unregistered [ 2600.584455] LNet: 153527:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2600.586609] LNetError: 153527:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2600.592400] LNet: Removed LNI 192.168.204.115@tcp [ 2600.905142] Key type .llcrypt unregistered [ 2600.906733] Key type ._llcrypt unregistered [ 2609.397096] Key type ._llcrypt registered [ 2609.397926] Key type .llcrypt registered [ 2609.431313] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2609.718977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2609.753051] alg: No test for adler32 (adler32-zlib) [ 2610.598853] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 2610.669139] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2612.248187] Key type lgssc registered [ 2612.592321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2614.855603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2617.106938] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 07:17:23 (1756811843) [ 2622.236574] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 2625.516818] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2628.799371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2631.085230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2633.039588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2635.147379] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2637.410908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2637.427671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2637.434522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2638.514681] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2638.526665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2638.566546] Lustre: lustre-MDT0000: new disk, initializing [ 2638.590032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2638.596558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2639.804117] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2643.861818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2643.887062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2643.921652] Lustre: 157719:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2643.945360] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2643.947635] Lustre: Skipped 1 previous similar message [ 2643.982587] Lustre: lustre-MDT0001: new disk, initializing [ 2644.017910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2644.031562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2644.035627] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2645.354445] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2648.296568] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2649.382552] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2651.453128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2651.477376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2651.579763] Lustre: lustre-OST0000: new disk, initializing [ 2651.581632] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2651.617627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2653.321075] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2655.351202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2655.355995] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2655.392994] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2656.292288] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2660.835164] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2660.835730] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2660.836144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2660.839021] Lustre: Skipped 1 previous similar message [ 2661.551091] LustreError: 160306:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2661.614808] Lustre: server umount lustre-OST0000 complete [ 2664.416948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2664.418668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2664.421821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2664.424018] Lustre: Skipped 1 previous similar message [ 2665.953213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2665.953857] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2668.987327] LustreError: 160506:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2668.988784] LustreError: 160506:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2669.016085] Lustre: server umount lustre-MDT0000 complete [ 2670.213262] LustreError: 158499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756811897 with bad export cookie 9420020644137948891 [ 2670.216257] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2670.218075] LustreError: 158499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2670.283059] LustreError: 160709:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2670.285616] LustreError: 160709:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2670.357539] Lustre: server umount lustre-MDT0001 complete [ 2673.835516] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2674.880054] Key type lgssc unregistered [ 2675.024327] LNet: 161240:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2675.026382] LNetError: 161240:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2675.037331] LNet: Removed LNI 192.168.204.115@tcp [ 2675.328986] Key type .llcrypt unregistered [ 2675.329980] Key type ._llcrypt unregistered [ 2684.126356] Key type ._llcrypt registered [ 2684.127487] Key type .llcrypt registered [ 2684.162080] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2684.456385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2684.487125] alg: No test for adler32 (adler32-zlib) [ 2685.329111] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 2685.407365] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2686.984241] Key type lgssc registered [ 2687.360346] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2690.051542] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2690.057971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2691.202037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2692.529447] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2695.214769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2695.318758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2696.580428] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2698.435147] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2699.629627] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2701.971711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2702.097175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2704.122671] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2705.991221] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2715.434963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2718.783523] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2718.791665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2718.797710] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2718.800901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2718.805533] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2719.140843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2719.525529] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2719.528278] Lustre: Skipped 1 previous similar message [ 2719.536537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2719.539882] Lustre: Skipped 1 previous similar message [ 2719.541497] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2719.543379] Lustre: Skipped 1 previous similar message [ 2719.544877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2719.549351] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2719.551483] Lustre: Skipped 1 previous similar message [ 2719.855461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2720.167638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2720.466966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2720.784546] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2720.787299] Lustre: Skipped 3 previous similar messages [ 2720.793921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2720.798657] Lustre: Skipped 3 previous similar messages [ 2720.800800] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2720.803383] Lustre: Skipped 3 previous similar messages [ 2720.805701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2720.809772] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2720.812604] Lustre: Skipped 3 previous similar messages [ 2721.134611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2721.499796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2721.847934] LustreError: 162631:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2721.852631] LustreError: 162631:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2721.853813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2722.204712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2722.520861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2722.845211] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2722.846993] Lustre: Skipped 5 previous similar messages [ 2722.853612] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2722.857454] Lustre: Skipped 5 previous similar messages [ 2722.859278] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2722.861405] Lustre: Skipped 5 previous similar messages [ 2722.862716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2722.866468] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2722.868703] Lustre: Skipped 5 previous similar messages [ 2723.215103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2723.533360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2723.878951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2724.197351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2724.564791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2724.877244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2725.206228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2725.559917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2725.900099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2726.207182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2726.552387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2726.885691] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2726.888511] Lustre: Skipped 11 previous similar messages [ 2726.895451] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2726.900358] Lustre: Skipped 11 previous similar messages [ 2726.902673] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2726.905179] Lustre: Skipped 11 previous similar messages [ 2726.906868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2726.910783] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2726.913151] Lustre: Skipped 11 previous similar messages [ 2727.272947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2727.621299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2727.956037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2728.295085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2728.627956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2728.955879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2729.270097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2729.573731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2729.909355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2730.227053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2730.534355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2730.855047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2731.182539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2731.496348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2731.801513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2732.108633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2732.426120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2732.731893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2733.032447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2733.346323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2733.713434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2734.040738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2734.366991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2734.677927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2735.031391] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2735.034400] Lustre: Skipped 24 previous similar messages [ 2735.040661] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2735.043967] Lustre: Skipped 24 previous similar messages [ 2735.045891] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2735.048776] Lustre: Skipped 24 previous similar messages [ 2735.050756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2735.053766] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2735.056303] Lustre: Skipped 24 previous similar messages [ 2735.378651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2735.682944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2735.984785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2736.317872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2736.639565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2736.959491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2737.267872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2737.604853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2737.920907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2738.264381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2738.620508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2738.990913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2739.344323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2739.685742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2740.061031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2740.418261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2740.759180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2741.087166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2741.409640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2741.737969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2742.058435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2742.408446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2742.723024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2743.050500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2743.396886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2743.747336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2744.073601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2744.431229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2744.764626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2745.134211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2745.488499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2745.824444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2746.152996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2746.513804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2746.827205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2747.155922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2747.466660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2747.808826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2748.123066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2748.447732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2748.793204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2749.159370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2749.493269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2749.817663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2750.155903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2750.500744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2750.800856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2751.096881] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2751.099328] Lustre: Skipped 47 previous similar messages [ 2751.105451] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2751.111168] Lustre: Skipped 47 previous similar messages [ 2751.113576] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2751.116948] Lustre: Skipped 47 previous similar messages [ 2751.119550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2751.123878] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2751.126060] Lustre: Skipped 47 previous similar messages [ 2751.422640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2751.754186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2752.070691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2752.374789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2752.684609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2753.030409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2753.359784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2753.680571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2753.980334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2754.294446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2754.604350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2754.912438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2755.222216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2755.551889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2755.870353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2756.177479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2756.477072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2756.805448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2757.115408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2757.420814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2757.735204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2758.069733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2758.393348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2758.711510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2759.021910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2759.341162] LustreError: 162631:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2759.343485] LustreError: 162631:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2759.345470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2759.655107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2759.968521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2760.264845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2760.572435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2760.881131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2761.190940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2761.517750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2761.844418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2762.154633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2762.465357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2762.755646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2763.085689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2763.385859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2763.685386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2764.002235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2764.330197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2764.647621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2764.965436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2765.288595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2765.627977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2765.972290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2766.295251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2766.615069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2766.961478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2767.288600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2767.627043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2767.958654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2768.327527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2768.669386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2768.994739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2769.328332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2769.688152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2770.007612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2770.336726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2770.657829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2770.997629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2771.323386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2771.639990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2771.960401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2772.300900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2772.613768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2772.944305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2773.276481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2773.640327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2773.976952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2774.347558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2774.697121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2775.022407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2775.368502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2775.714743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2776.071969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2776.407477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2776.742176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2777.064963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2777.422713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2777.764161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2778.097587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2778.428863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2778.792627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2779.133126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2779.441996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2779.745128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2780.069156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2780.369897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2780.675252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2780.976420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2781.283842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2781.576585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2781.867317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2782.155617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2782.442922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2782.727739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2783.019105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2783.303594] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2783.306484] Lustre: Skipped 99 previous similar messages [ 2783.311693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2783.315569] Lustre: Skipped 99 previous similar messages [ 2783.317285] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2783.320373] Lustre: Skipped 99 previous similar messages [ 2783.322559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2783.326229] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2783.329411] Lustre: Skipped 99 previous similar messages [ 2783.617083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2783.903086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2784.221639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2784.515246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2784.814891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2785.109157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2785.397537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2785.682815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2785.973354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2786.259802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2786.555072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2786.846892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2787.165046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2787.473309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2787.777085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2788.083276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2788.402128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2788.688570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2788.996285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2789.285751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2789.594715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2789.902359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2790.213022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2790.508584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2790.827394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2791.127383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2791.423409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2791.717961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2792.031075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2792.311703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2792.616345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2792.918910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2793.240667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2793.535655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2793.843578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2794.136383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2794.450635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2794.734509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2795.034875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2795.335556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2795.658475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2795.977506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2796.302247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2796.610061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2796.934391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2797.246981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2797.550513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2797.826765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2798.125213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2798.420943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2798.722730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2799.022582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2799.340311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2799.628698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2799.907262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2800.197976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2800.507272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2800.808770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2801.109300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2801.415370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2801.737913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2802.036318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2802.349330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2802.636152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2802.942557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2803.231926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2803.512932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2803.791968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2804.073972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2804.368113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2804.663586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2804.962399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2805.283710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2805.580320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2805.874805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2806.175209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2806.482126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2806.779908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2807.079805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2807.389202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2807.692203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2807.986174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2808.292708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2808.588792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2808.910431] LustreError: 162631:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2808.914563] LustreError: 162631:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2808.915583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2809.213204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2809.508577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2809.803191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2810.134438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2810.458849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2810.753283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2811.060466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2811.379864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2811.712149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2812.012417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2812.304228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2812.601088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2812.893902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2813.177725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2813.464279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2813.746965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2814.049115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2814.332738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2814.599026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2814.866029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2815.157776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2815.433947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2815.696737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2815.957821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2816.260313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2816.558377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2816.857220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2817.163515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2817.480273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2817.783689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2818.095194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2818.390301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2818.716783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2819.043084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2819.374458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2819.685713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2820.024767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2820.351374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2820.676241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2820.998084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2821.337210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2821.645966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2821.979719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2822.296813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2822.636805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2822.942507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2823.278357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2823.606839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2823.965652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2824.283485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2824.615815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2824.931611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2825.280308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2825.603694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2825.927747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2826.241361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2826.584590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2826.912556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2827.222181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2827.537829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2827.888436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2828.214355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2828.509985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2828.843219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2829.198477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2829.533016] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2829.864127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2830.195090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2830.548808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2830.863967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2831.180370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2831.485602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2831.812202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2832.151586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2832.474307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2832.791642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2833.148351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2833.479300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2833.800357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2834.132143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2834.493978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2834.827922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2835.170369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2835.506514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2835.881569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2836.219749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2836.550301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2836.886687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2837.269913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2837.603517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2837.922725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2838.243464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2838.585425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2838.906779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2839.231829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2839.559933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2839.918344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2840.270173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2840.590225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2840.891969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2841.232040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2841.550772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2841.880185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2842.211966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2842.551340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2842.870604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2843.178919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2843.503043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2843.826382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2844.149821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2844.455179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2844.742288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2845.063518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2845.385312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2845.694150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2846.020990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2846.347706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2846.670722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2846.996195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2847.311762] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2847.314185] Lustre: Skipped 204 previous similar messages [ 2847.320133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2847.324566] Lustre: Skipped 204 previous similar messages [ 2847.326147] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2847.328354] Lustre: Skipped 204 previous similar messages [ 2847.331214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2847.336581] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2847.339044] Lustre: Skipped 204 previous similar messages [ 2847.675612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2847.991876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2848.300950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2848.611406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2848.954739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2849.296680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2849.612163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2849.948323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2850.282028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2850.590159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2850.904509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2851.221418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2851.549166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2851.876821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2852.204344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2852.498762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2852.819771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2853.154396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2853.466960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2853.775108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2854.109127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2854.405215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2854.710093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2855.011652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2855.315026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2855.609027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2855.906751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2856.221601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2856.541185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2856.868134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2857.188764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2857.509474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2857.842820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2858.156506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2858.466534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2858.790871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2859.135954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2859.454759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2859.772026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2860.085411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2860.410319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2860.722795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2861.032331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2861.358237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2861.697240] LustreError: 162631:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2861.702148] LustreError: 162631:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2861.702594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2862.028094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2862.346104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2862.665457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2862.994185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2863.323448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2863.643161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2863.965350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2864.296171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2864.618450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2864.930512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2865.252105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2865.586518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2865.910883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2866.218918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2866.527569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2866.858836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2867.168285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2867.476709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2867.780256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2868.120447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2868.426057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2868.727260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2869.063190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2869.400209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2869.732364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2870.058804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2870.374332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2870.721800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2871.061432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2871.372531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2871.687757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2872.007472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2872.343239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2872.662174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2872.978114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2873.314814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2873.640103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2873.965788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2874.294136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2874.666724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2874.986827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2875.307927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2875.644523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2875.980024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2876.290914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2876.610128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2876.923887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2877.259945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2877.569552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2877.881118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2878.191458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2878.531136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2878.858558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2879.170816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2879.484425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2879.807781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2880.122644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2880.459107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2880.778777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2881.116499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2881.432017] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2881.731876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2882.043677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2882.362791] LustreError: 162631:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2882.366842] LustreError: 162631:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2882.367377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2882.684264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2883.024832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2883.345864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2883.668106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2883.957177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2884.275047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2884.586854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2884.921506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2885.251098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2885.573601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2885.892465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2886.221873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2886.526182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2886.845466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2887.157109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2887.487059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2887.786260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2888.114713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2888.444785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2888.793856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2889.113949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2889.442074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2889.763789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2890.112358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2890.436500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2890.757570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2891.080667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2891.415958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2891.729971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2892.045572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2892.356217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2892.682590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2892.985489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2893.298915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2893.610953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2893.946470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2894.269451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2894.597951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2894.916762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2895.227079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2895.539198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2895.856396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2896.175185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2896.533349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2896.853416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2897.190726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2897.497485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2897.827178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2898.155690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2898.469889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2898.782475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2899.129084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2899.478606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2899.816065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2900.121368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2900.478570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2900.809269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2901.131476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2901.465594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2901.800903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2902.108639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2902.430410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2902.738877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2903.077544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2903.382454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2903.691697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2904.010509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2904.332921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2904.647549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2904.954348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2905.268954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2905.595941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2905.917581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2906.239536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2906.573827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2906.904401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2907.234774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2907.542971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2907.853065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2908.174127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2908.477927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2908.775224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2909.079846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2909.382928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2909.675442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2909.965973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2910.260969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2910.576194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2910.886167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2911.189805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2911.501379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2911.826465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2912.152416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2912.470139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2912.780447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2913.119359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2913.429370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2913.742327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2914.077895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2914.406602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2914.725178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2915.039467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2915.342969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2915.671735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2915.986556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2916.293738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2916.595725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2916.904763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2917.227880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2917.544103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2917.867243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2918.192706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2918.510666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2918.831851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2919.150018] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2919.463196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2919.778801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2920.095016] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2920.410202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2920.734809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2921.040541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2921.351846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2921.666655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2921.998173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2922.318417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2922.637696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2922.945890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2923.269417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2923.588357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2923.915679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2924.234782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2924.567295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2924.880759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2925.186038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2925.504133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2925.838987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2926.160041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2926.479294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2926.788820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2927.121270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2927.433391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2927.750670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2928.057865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2928.377860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2928.692245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2929.014779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2929.330915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2929.679457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2930.018600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2930.337284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2930.649223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2931.004886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2931.324754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2931.634069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2931.935796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2932.267918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2932.598316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2932.925258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2933.250515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2933.565052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2933.886434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2934.210118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2934.527781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2934.868121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2935.204852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2935.541093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2935.863836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2936.216096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2936.526160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2936.816343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2937.134521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2937.470643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2937.781223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2938.107890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2938.423059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2938.779427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2939.095197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2939.407252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2939.705218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2940.020696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2940.342147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2940.623813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2940.885132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2941.167031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2941.445468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2941.723251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2941.992547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2942.281646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2942.571747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2942.877922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2943.188453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2943.475274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2943.741532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2944.011571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2944.291362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2944.567306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2944.825303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2945.103029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2945.389446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2945.686127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2945.978959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2946.288730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2946.582739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2946.907283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2947.205861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2947.507164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2947.809524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2948.145317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2948.484891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2948.797350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2949.092226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2949.407493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2949.697680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2950.007615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2950.296515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2950.607716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2950.912631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2951.218225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2951.509830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2951.809765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2952.093528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2952.405606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2952.716816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2953.047702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2953.330520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2953.647190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2953.971956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2954.300372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2954.610680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2954.896407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2955.183940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2955.471238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2955.776839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2956.068615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2956.360832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2956.670707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2956.978942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2957.296048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2957.613214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2957.952989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2958.257849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2958.547963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2958.816281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2959.107570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2959.401548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2959.703761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2960.003190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2960.303126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2960.609788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2960.938412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2961.237848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2961.543155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2961.841657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2962.126605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2962.403367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2962.690771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2962.967183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2963.244645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2963.525399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2963.829781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2964.116692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2964.401881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2964.684343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2964.979398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2965.255556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2965.546277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2965.824073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2966.108333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2966.391904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2966.666393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2966.943158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2967.216527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2967.495690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2967.772378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2968.057593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2968.342665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2968.623715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2968.900173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2969.175735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2969.465891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2969.737647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2970.020235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2970.287185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2970.551494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2970.824268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2971.122503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2971.433921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2971.747611] LustreError: 162631:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2971.750450] LustreError: 162631:0:(osp_precreate.c:651:osp_precreate_send()) Skipped 1 previous similar message [ 2971.752467] LustreError: 162631:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2971.753140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2971.755458] LustreError: 162631:0:(osp_precreate.c:1355:osp_precreate_thread()) Skipped 1 previous similar message [ 2972.090132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2972.397791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2972.700339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2973.036648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2973.351783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2973.673544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2973.981557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2974.288391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2974.593398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2974.897715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2975.204847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2975.509519] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2975.511656] Lustre: Skipped 408 previous similar messages [ 2975.517314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2975.520603] Lustre: Skipped 408 previous similar messages [ 2975.521963] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2975.523868] Lustre: Skipped 408 previous similar messages [ 2975.525217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2975.529502] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2975.532847] Lustre: Skipped 408 previous similar messages [ 2975.820555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2976.109768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2976.398182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2976.702460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2976.996652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2977.315313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2977.652664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2977.982224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2978.305966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2978.623730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2978.921813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2979.232063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2979.524469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2979.819282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2980.110410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2980.419474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2980.719268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2981.013650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2981.322295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2981.644934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2981.954218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2982.258711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2982.561644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2982.868067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2983.160294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2983.445252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2983.738772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2984.053485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2984.362369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2984.650729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2984.962732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2985.270816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2985.579743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2985.884699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2986.202152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2986.533227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2986.857520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2987.157392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2987.492199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2987.837337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2988.144673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2988.448779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2988.743942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2989.061383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2989.378668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2989.705532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2990.033818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2990.373374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2990.672346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2990.973806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2991.280434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2991.601658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2991.900362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2992.202336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2992.513458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2992.825975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2993.133864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2993.425277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2993.739459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2994.063608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2994.361038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2994.661317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2994.963382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2995.282768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2995.573227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2995.867954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2996.170829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2996.497654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2996.810860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2997.127221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2997.448558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2997.812981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2998.135180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2998.467052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2998.795275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2999.136507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2999.450628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2999.763090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3000.090958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3000.430686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3000.737590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3001.058790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3001.375874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3001.705711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3002.013888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3002.334538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3002.651540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3002.979119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3003.308664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3003.638111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3003.967526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3004.310642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3004.646343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3004.973347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3005.310910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3005.665648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3005.946930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3006.236208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3006.528537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3006.835376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3007.161109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3007.452105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3007.744516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3008.076051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3008.389272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3008.699518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3008.984827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3009.274586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3009.551200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3009.829322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3010.100184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3010.380563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3010.661777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3010.946192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3011.231051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3011.548920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3011.839994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3012.123725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3012.417506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3012.712280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3012.998226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3013.287396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3013.567336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3013.865902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3014.161631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3014.444106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3014.717361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3014.996971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3015.281671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3015.559442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3015.839454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3016.111475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3016.385457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3016.700930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3016.989347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3017.296339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3017.579964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3017.867127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3018.159902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3018.482092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3018.796185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3019.086669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3019.390396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3019.699597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3019.992059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3020.322711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3020.624491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3020.925349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3021.221120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3021.515925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3021.794388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3022.071642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3022.343822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3022.619493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3022.897295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3023.191322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3023.476701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3023.760067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3024.075481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3024.384259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3024.662497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3024.948662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3025.230240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3025.522047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3025.801723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3026.092948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3026.377296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3026.668513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3026.963113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3027.273671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3027.596172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3027.897357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3028.192280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3028.481173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3028.798467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3029.137155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3029.428771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3029.710712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3030.002349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3030.302890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3030.615394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3030.910295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3031.236426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3031.579884] LustreError: 162631:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 3031.582784] LustreError: 162631:0:(osp_precreate.c:651:osp_precreate_send()) Skipped 1 previous similar message [ 3031.585134] LustreError: 162631:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 3031.586275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3031.587548] LustreError: 162631:0:(osp_precreate.c:1355:osp_precreate_thread()) Skipped 1 previous similar message [ 3031.914772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3032.226224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3032.541319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3032.862055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3089.416748] LustreError: 164209:0:(ofd_dev.c:1776:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 3089.420158] LustreError: 162631:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 3151.840699] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3151.844489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3155.424899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3155.430519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3158.017731] LustreError: 216355:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3158.262312] Lustre: server umount lustre-OST0000 complete [ 3161.060049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3161.062232] Lustre: Skipped 2 previous similar messages [ 3165.760479] LustreError: 216557:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3165.762381] LustreError: 216557:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3165.846437] Lustre: server umount lustre-MDT0000 complete [ 3166.177189] LustreError: 162611:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3166.183115] LustreError: 162611:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 3167.046085] LustreError: 164230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756812393 with bad export cookie 10233423569115476104 [ 3167.048938] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3167.050043] LustreError: 164230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3167.118958] LustreError: 216758:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3167.122070] LustreError: 216758:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3167.200873] Lustre: server umount lustre-MDT0001 complete [ 3170.922022] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 3172.053260] Key type lgssc unregistered [ 3172.195391] LNet: 217288:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3172.199282] LNetError: 217288:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3172.209327] LNet: Removed LNI 192.168.204.115@tcp [ 3172.528254] Key type .llcrypt unregistered [ 3172.529461] Key type ._llcrypt unregistered [ 3181.704141] Key type ._llcrypt registered [ 3181.705449] Key type .llcrypt registered [ 3181.740370] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3182.065283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3182.072240] alg: No test for adler32 (adler32-zlib) [ 3182.934626] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 3183.012917] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 3184.592138] Key type lgssc registered [ 3184.948295] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3187.462565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3187.469906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3188.626620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3189.966713] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3192.895808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3193.035282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3194.459183] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3196.241651] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3197.499661] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3199.803196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3199.906444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3201.745819] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3203.501225] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3214.249642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49666 to 0x280000401:49697) [ 3221.854577] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3222.451980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3222.455913] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3222.459648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3222.463498] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3222.463521] LustreError: 218695:0:(osp_precreate.c:1263:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 3223.651431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3223.651860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3223.654218] LustreError: Skipped 2 previous similar messages [ 3223.662321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3224.332989] Lustre: lustre-OST0000: Not available for connect from 192.168.204.15@tcp (stopping) [ 3225.708182] LustreError: 221097:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3225.778695] Lustre: server umount lustre-OST0000 complete [ 3242.544130] LustreError: 218695:0:(osp_precreate.c:1263:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 3253.813092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3253.946557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3255.904965] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3257.777875] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3265.189420] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3265.194960] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3265.531530] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 7 sec [ 3266.693833] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff97502a943800.ost_server_uuid 50 [ 3267.191824] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff97502a943800.ost_server_uuid in FULL state after 0 sec [ 3270.625979] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3270.626145] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3270.630236] Lustre: Skipped 1 previous similar message [ 3270.630736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3270.633819] LustreError: Skipped 2 previous similar messages [ 3270.638618] Lustre: Skipped 2 previous similar messages [ 3274.284188] LustreError: 222419:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3274.286404] LustreError: 222419:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3274.341377] Lustre: server umount lustre-OST0000 complete [ 3279.841597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3279.842370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3279.845765] Lustre: Skipped 1 previous similar message [ 3279.849762] Lustre: Skipped 1 previous similar message [ 3281.730179] LustreError: 222620:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3281.732938] LustreError: 222620:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3281.757404] Lustre: server umount lustre-MDT0000 complete [ 3283.062166] LustreError: 218655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756812509 with bad export cookie 12162021086668452413 [ 3283.065204] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3283.067676] LustreError: 218655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3283.215958] Lustre: server umount lustre-MDT0001 complete [ 3286.784114] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 3287.869796] Key type lgssc unregistered [ 3288.004447] LNet: 223351:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3288.008550] LNetError: 223351:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3288.019324] LNet: Removed LNI 192.168.204.115@tcp [ 3288.348577] Key type .llcrypt unregistered [ 3288.349709] Key type ._llcrypt unregistered [ 3299.721338] Key type ._llcrypt registered [ 3299.722356] Key type .llcrypt registered [ 3299.759951] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3300.102551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3300.168545] alg: No test for adler32 (adler32-zlib) [ 3301.021847] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 3301.111854] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 3302.688172] Key type lgssc registered [ 3303.072503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3306.109048] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3306.114925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3307.268032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3308.534525] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3311.311388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3311.427610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3312.719450] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3313.673375] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3315.859982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3315.982205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3317.913500] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3318.057608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49699 to 0x280000401:49729) [ 3323.197072] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3323.255582] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3323.509025] Lustre: lustre-OST0001: new disk, initializing [ 3323.513658] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3323.667457] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3327.269344] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3328.567697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3328.572601] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3328.633115] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3334.042871] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3336.649685] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3349.476760] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3349.478766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3349.482437] Lustre: Skipped 3 previous similar messages [ 3349.485070] Lustre: Skipped 2 previous similar messages [ 3354.111517] LustreError: 228401:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3354.155871] Lustre: server umount lustre-MDT0000 complete [ 3354.593687] LustreError: 225749:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3354.600241] LustreError: 225749:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 5 previous similar messages [ 3355.456382] LustreError: 225018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756812582 with bad export cookie 1520673617324992541 [ 3355.459570] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3355.459958] LustreError: 225018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3355.531797] LustreError: 228603:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3355.533957] LustreError: 228603:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3355.608924] Lustre: server umount lustre-MDT0001 complete [ 3366.901182] LustreError: 228803:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3366.903304] LustreError: 228803:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3366.927695] Lustre: server umount lustre-OST0000 complete [ 3377.396159] LustreError: 229006:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3377.398529] LustreError: 229006:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3377.465449] Lustre: server umount lustre-OST0001 complete [ 3378.519713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3378.551912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3378.893962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3378.918085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3379.251490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3379.269060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3379.606229] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3379.635476] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3384.212887] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3387.544296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3387.718841] LustreError: 230412:0:(ldlm_lib.c:1132:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3387.744030] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3389.123884] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3392.075054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3393.521579] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3394.509749] Lustre: 231522:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3396.840220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3396.989131] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3396.991646] Lustre: Skipped 1 previous similar message [ 3397.988473] LustreError: 231877:0:(ldlm_lib.c:1132:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3397.995741] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49732 to 0x280000401:49761) [ 3399.867436] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3402.083682] LustreError: 232051:0:(ldlm_lib.c:1132:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3402.092595] LustreError: 232051:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 3402.795690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3404.769601] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3407.851829] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3408.991135] Lustre: 233370:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3418.594900] Lustre: mylustre-MDT0000-lwp-OST0000: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3418.596478] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3418.599271] Lustre: Skipped 2 previous similar messages [ 3418.600869] Lustre: Skipped 1 previous similar message [ 3423.201099] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3423.203216] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3423.208323] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3424.381758] LustreError: 233679:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3424.383662] LustreError: 233679:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3424.412059] Lustre: server umount mylustre-MDT0000 complete [ 3425.599141] LustreError: 233373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756812652 with bad export cookie 1520673617325080104 [ 3425.603345] LustreError: 233373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3425.603504] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3425.745277] Lustre: server umount mylustre-MDT0001 complete [ 3437.042226] LustreError: 234083:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3437.044158] LustreError: 234083:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 3437.061602] Lustre: server umount mylustre-OST0000 complete [ 3448.624422] Lustre: server umount mylustre-OST0001 complete [ 3449.674951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3449.707324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3450.084293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3450.113324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3450.445202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3450.469170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3450.815722] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3450.844837] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3455.725196] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3459.306855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3459.464380] LustreError: 235693:0:(ldlm_lib.c:1132:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3459.485447] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3459.488758] Lustre: Skipped 1 previous similar message [ 3460.800277] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3463.646969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3465.044087] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3466.013156] Lustre: 236800:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3468.059753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3469.221890] LustreError: 237348:0:(ldlm_lib.c:1132:target_handle_connect()) tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3469.225930] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49732 to 0x280000401:49793) [ 3470.000538] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3472.841883] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3473.966745] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3474.792191] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3477.871308] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3479.046542] Lustre: 238650:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3489.255360] Lustre: tfs-MDT0000-lwp-OST0000: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3489.256482] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3489.257897] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 3489.257904] Lustre: Skipped 3 previous similar messages [ 3489.258617] Lustre: Skipped 1 previous similar message [ 3492.096651] LustreError: 238899:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3492.099617] LustreError: 238899:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3492.129934] Lustre: server umount tfs-MDT0000 complete [ 3493.462298] LustreError: 235669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756812720 with bad export cookie 1520673617325081770 [ 3493.466838] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3493.466944] LustreError: 235669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3493.606568] Lustre: server umount tfs-MDT0001 complete [ 3505.154624] Lustre: server umount tfs-OST0000 complete [ 3516.719831] Lustre: server umount tfs-OST0001 complete [ 3517.722696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3517.748462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3518.087262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3518.112135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3518.432743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3518.454611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3518.795380] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3518.826335] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3523.305959] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3526.623507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3526.767659] LustreError: 240909:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3526.789761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3526.791595] Lustre: Skipped 3 previous similar messages [ 3528.066085] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3530.938411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3532.375840] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3533.331334] Lustre: 242022:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3535.636830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3536.809227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49732 to 0x280000401:49825) [ 3537.692856] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3540.516599] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3541.611457] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3542.548989] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3545.646830] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3546.817509] Lustre: 243869:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3554.138298] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 07:33:00 (1756812780) [ 3567.072859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3567.076093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3567.081501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3567.083214] Lustre: Skipped 3 previous similar messages [ 3577.826646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3577.828451] Lustre: Skipped 10 previous similar messages [ 3580.384107] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3580.418534] LustreError: 244236:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3580.420929] LustreError: 244236:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3580.450334] Lustre: server umount lustre-MDT0000 complete [ 3581.568095] LustreError: 243871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756812808 with bad export cookie 1520673617325083436 [ 3581.570728] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3581.572686] LustreError: 243871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3582.945261] LustreError: 240904:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3582.945889] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3582.949507] LustreError: 240904:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 3582.954189] Lustre: Skipped 4 previous similar messages [ 3587.792728] Lustre: server umount lustre-MDT0001 complete [ 3595.195657] Lustre: server umount lustre-OST0000 complete [ 3602.605256] Lustre: server umount lustre-OST0001 complete [ 3604.599818] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 3607.807528] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3611.462486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3613.661926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3615.674839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3617.666536] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3619.884187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3619.905384] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 3619.905384] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 3619.905384] [ 3619.911362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 3619.992826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3620.004390] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3620.043214] Lustre: lustre-MDT0000: new disk, initializing [ 3620.067091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3620.069499] Lustre: Skipped 3 previous similar messages [ 3620.074880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3621.372327] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3625.417719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3625.441358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3625.473544] Lustre: 247023:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3625.497723] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3625.499420] Lustre: Skipped 1 previous similar message [ 3625.533805] Lustre: lustre-MDT0001: new disk, initializing [ 3625.581606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3625.586407] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3626.914041] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3631.139208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3631.163572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3631.263971] Lustre: lustre-OST0000: new disk, initializing [ 3631.265890] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3633.153827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3633.158443] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3633.209940] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3633.296630] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3637.532987] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3637.557210] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3637.613231] Lustre: lustre-OST0001: new disk, initializing [ 3637.615579] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3638.975464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3638.980237] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3639.016382] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3639.682476] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3643.973847] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 07:34:30 (1756812870) [ 3647.595825] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 07:34:33 (1756812873) [ 3659.748652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3659.750084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3659.753424] Lustre: Skipped 1 previous similar message [ 3659.755644] Lustre: Skipped 4 previous similar messages [ 3665.133220] LustreError: 250447:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3665.136698] LustreError: 250447:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3665.157155] Lustre: server umount lustre-OST0000 complete [ 3669.986936] LustreError: 248564:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3669.991944] LustreError: 248564:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 3672.669610] Lustre: server umount lustre-MDT0000 complete [ 3673.914539] LustreError: 247016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756812900 with bad export cookie 1520673617325084990 [ 3673.916983] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3673.920200] LustreError: 247016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3680.138969] Lustre: server umount lustre-MDT0001 complete [ 3683.839533] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 3693.620342] Lustre: server umount lustre-OST0001 complete [ 3695.678799] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 3702.126069] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3705.754546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3708.026381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3710.039274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3712.121279] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3714.385941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3714.406496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3714.492627] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3714.506288] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3714.547950] Lustre: lustre-MDT0000: new disk, initializing [ 3714.580982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3715.977140] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3720.073857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3720.096185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3720.131182] Lustre: 253905:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3720.247036] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3721.693954] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3724.890810] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3726.176566] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3728.409854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3728.436394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3730.361578] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3730.406028] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3730.544192] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3733.702337] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3740.643546] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3740.644159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3740.647941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3740.653576] Lustre: Skipped 5 previous similar messages [ 3740.655065] Lustre: Skipped 8 previous similar messages [ 3742.186164] Lustre: server umount lustre-OST0000 complete [ 3745.762039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3745.764603] LustreError: Skipped 1 previous similar message [ 3749.525138] Lustre: server umount lustre-MDT0000 complete [ 3750.754433] LustreError: 253895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756812977 with bad export cookie 1520673617325086460 [ 3750.757321] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3750.759783] LustreError: 253895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3750.908339] Lustre: server umount lustre-MDT0001 complete [ 3754.663526] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 3755.801083] Key type lgssc unregistered [ 3755.954521] LNet: 257279:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3755.962468] LNetError: 257279:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3755.975421] LNet: Removed LNI 192.168.204.115@tcp [ 3756.371138] Key type .llcrypt unregistered [ 3756.372526] Key type ._llcrypt unregistered [ 3766.381273] Key type ._llcrypt registered [ 3766.383261] Key type .llcrypt registered [ 3766.424474] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3766.897566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3766.917038] alg: No test for adler32 (adler32-zlib) [ 3767.784459] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 3767.867880] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 3769.456140] Key type lgssc registered [ 3769.824038] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3772.408568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3772.414695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3773.557283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3774.880850] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3777.742713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3777.863478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3779.277862] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3781.094646] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3782.280668] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3784.501941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3784.605531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3786.394699] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3788.108746] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3788.427065] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 3792.865563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3792.867390] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3792.873337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3795.116284] LustreError: 260990:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3795.181592] Lustre: server umount lustre-OST0000 complete [ 3796.416231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3796.422992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3796.429426] Lustre: Skipped 1 previous similar message [ 3801.569758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3801.572398] Lustre: Skipped 1 previous similar message [ 3802.558180] LustreError: 261205:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3802.561476] LustreError: 261205:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3802.588035] Lustre: server umount lustre-MDT0000 complete [ 3803.875502] LustreError: 258680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756813030 with bad export cookie 6976540315635126332 [ 3803.879520] LustreError: 258680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3803.881318] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3803.950580] LustreError: 261406:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3803.954068] LustreError: 261406:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3804.060402] Lustre: server umount lustre-MDT0001 complete [ 3807.625919] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 3808.591494] Key type lgssc unregistered [ 3808.722344] LNet: 261937:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3808.724422] LNetError: 261937:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3808.733287] LNet: Removed LNI 192.168.204.115@tcp [ 3809.048165] Key type .llcrypt unregistered [ 3809.049120] Key type ._llcrypt unregistered [ 3815.354570] Key type ._llcrypt registered [ 3815.356248] Key type .llcrypt registered [ 3815.392359] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3815.836528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3815.844947] alg: No test for adler32 (adler32-zlib) [ 3816.696904] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 3816.773734] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 3818.360148] Key type lgssc registered [ 3818.697970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3821.545858] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3821.552307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3839.008338] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3845.152185] LustreError: 263087:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.115@tcp: failed processing log, type 4: rc = -110 [ 3875.872194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3877.704726] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3879.464804] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3880.172612] LustreError: 263751:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3880.249262] Lustre: server umount lustre-OST0000 complete [ 3885.797918] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 3886.870043] Key type lgssc unregistered [ 3887.002511] LNet: 264589:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3887.005222] LNetError: 264589:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3887.013423] LNet: Removed LNI 192.168.204.115@tcp [ 3887.351838] Key type .llcrypt unregistered [ 3887.353677] Key type ._llcrypt unregistered [ 3900.535725] Key type ._llcrypt registered [ 3900.536823] Key type .llcrypt registered [ 3900.567784] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3900.977106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3900.992922] alg: No test for adler32 (adler32-zlib) [ 3901.848486] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 3901.924128] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 3903.504130] Key type lgssc registered [ 3903.846875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3908.169100] loop0: detected capacity change from 0 to 204800000 [ 3908.225849] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3908.230319] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3908.346962] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3908.351215] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3908.373561] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3908.393368] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3908.409948] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3908.434254] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3908.462174] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3908.491851] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3908.821497] loop0: detected capacity change from 0 to 204800000 [ 3908.887309] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3909.135259] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3909.503178] loop0: detected capacity change from 0 to 204800000 [ 3909.638197] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3909.960556] loop0: detected capacity change from 0 to 204800000 [ 3910.104018] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3910.445689] loop0: detected capacity change from 0 to 204800000 [ 3910.459631] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3911.717739] loop0: detected capacity change from 0 to 204800000 [ 3911.725591] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3913.005391] loop0: detected capacity change from 0 to 204800000 [ 3913.020105] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3914.311700] loop0: detected capacity change from 0 to 204800000 [ 3914.323434] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3915.569845] loop0: detected capacity change from 0 to 204800000 [ 3915.602538] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3915.620922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3915.626634] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3915.649587] Lustre: 267963:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3915.654113] Lustre: 267963:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 3915.657776] Lustre: 267963:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 3915.661321] Lustre: 267963:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3915.664958] Lustre: 267963:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3915.668143] Lustre: 267963:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 3915.671444] Lustre: 267963:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3915.674982] Lustre: 267963:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 3915.678944] Lustre: 267963:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 3915.682743] Lustre: 267963:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 3915.686423] Lustre: 267963:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 3915.689587] Lustre: 267963:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 3915.692505] Lustre: 267963:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 3915.695514] Lustre: 267963:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 3918.138147] LustreError: 268230:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3918.178843] Lustre: server umount MGS complete [ 3918.456787] loop0: detected capacity change from 0 to 204800000 [ 3918.493390] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3918.655347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3918.671424] Lustre: lustre-MDT0000: Aborting client recovery [ 3918.955626] loop1: detected capacity change from 0 to 204800000 [ 3919.004991] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3919.033230] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3919.056069] Lustre: 268400:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3919.059138] Lustre: 268400:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3919.062730] Lustre: 268400:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3919.066854] Lustre: 268400:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 3919.070603] Lustre: 268400:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3919.074131] Lustre: 268400:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 3919.077141] Lustre: 268400:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 3919.083783] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3919.177270] Lustre: lustre-MDT0001: Aborting client recovery [ 3919.484591] loop2: detected capacity change from 0 to 204800000 [ 3919.518569] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3919.567524] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3919.585441] Lustre: 268484:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3919.588726] Lustre: 268484:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3919.591875] Lustre: 268484:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3919.595387] Lustre: 268484:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3919.599616] Lustre: 268484:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3919.602772] Lustre: 268484:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3919.605734] Lustre: 268484:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3919.656146] Lustre: Found index 0 for lustre-OST0000, updating log [ 3919.684302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3919.686017] Lustre: Skipped 1 previous similar message [ 3919.690758] Lustre: lustre-OST0000: Aborting recovery [ 3919.991487] loop3: detected capacity change from 0 to 204800000 [ 3920.059223] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3920.083951] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3920.103907] Lustre: 268572:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3920.109099] Lustre: 268572:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3920.114761] Lustre: 268572:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3920.120206] Lustre: 268572:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3920.125850] Lustre: 268572:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3920.131434] Lustre: 268572:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3920.136848] Lustre: 268572:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3925.481292] LustreError: 268342:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -108 [ 3928.549183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 3928.549970] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 3928.554840] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 3928.558234] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 3928.560584] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 3928.563239] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 3930.869114] LustreError: 268309:0:(mdt_identity.c:95:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 220us: rc = -2 [ 3943.905856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3943.906227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3943.909566] Lustre: Skipped 2 previous similar messages [ 3943.915122] Lustre: Skipped 3 previous similar messages [ 3948.287277] LustreError: 268929:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3948.289227] LustreError: 268929:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3948.329573] Lustre: server umount lustre-MDT0000 complete [ 3948.637930] LustreError: 268292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756813175 with bad export cookie 18433659187229740806 [ 3948.639858] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3948.642033] LustreError: 268292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3949.024642] LustreError: 268307:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3949.024829] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3949.025057] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3949.033078] LustreError: 268307:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 3949.039462] Lustre: Skipped 2 previous similar messages [ 3953.056543] LustreError: 268307:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3953.056682] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3953.062638] LustreError: 268307:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 3953.064677] Lustre: Skipped 2 previous similar messages [ 3954.812625] LustreError: 268980:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3954.815164] LustreError: 268980:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3954.908394] Lustre: server umount lustre-MDT0001 complete [ 3961.322060] LustreError: 269031:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3961.324423] LustreError: 269031:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3961.344771] Lustre: server umount lustre-OST0000 complete [ 3967.723853] LustreError: 269081:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3967.725453] LustreError: 269081:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3967.794613] Lustre: server umount lustre-OST0001 complete [ 3970.404680] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 07:39:56 (1756813196) [ 3979.194064] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 3982.293579] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3985.715692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3987.852361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3989.768404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3991.738246] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3993.920667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3993.940967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3994.019832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3994.031109] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3994.067254] Lustre: lustre-MDT0000: new disk, initializing [ 3994.091653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3994.093942] Lustre: Skipped 1 previous similar message [ 3994.099211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3995.364750] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3999.349565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3999.372115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3999.402223] Lustre: 272625:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3999.424352] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3999.426308] Lustre: Skipped 1 previous similar message [ 3999.459885] Lustre: lustre-MDT0001: new disk, initializing [ 3999.490949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3999.501780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3999.504428] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4000.730387] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4003.697068] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4004.886600] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4007.100185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4007.124299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4007.218804] Lustre: lustre-OST0000: new disk, initializing [ 4007.220497] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4007.250704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4009.077417] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4012.097143] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4017.147120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4017.151265] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4017.185895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4019.728365] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 4027.414113] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 4034.474989] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 4044.050044] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 4088.803605] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4088.803938] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4088.805604] LustreError: Skipped 1 previous similar message [ 4088.806180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4088.810177] Lustre: Skipped 1 previous similar message [ 4091.055127] LustreError: 276339:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4091.058846] LustreError: 276339:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4091.139462] Lustre: server umount lustre-OST0000 complete [ 4093.921840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4093.922469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4093.926788] Lustre: Skipped 1 previous similar message [ 4093.928984] Lustre: Skipped 1 previous similar message [ 4098.581939] Lustre: server umount lustre-MDT0000 complete [ 4099.041944] LustreError: 272634:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4099.046647] LustreError: 272634:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 4099.789117] LustreError: 272616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756813326 with bad export cookie 18433659187229743522 [ 4099.792648] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4099.794356] LustreError: 272616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4099.936467] Lustre: server umount lustre-MDT0001 complete [ 4102.735153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4103.925851] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4105.479653] Lustre: server umount MGS complete [ 4109.201262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4109.335979] LustreError: 278304:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4109.341939] LustreError: 278316:0:(qmt_pool.c:1513:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 4109.363408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4110.562930] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4113.250430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4114.740392] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4116.499941] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4117.662906] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4119.831157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4121.741994] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4123.372448] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4159.970923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4159.971980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4159.972958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4159.972963] Lustre: Skipped 1 previous similar message [ 4159.973537] LustreError: Skipped 1 previous similar message [ 4159.976987] Lustre: Skipped 1 previous similar message [ 4165.997121] LustreError: 280985:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4165.999472] LustreError: 280985:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 4166.062874] Lustre: server umount lustre-OST0000 complete [ 4169.697070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4169.700617] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4173.394799] Lustre: server umount lustre-MDT0000 complete [ 4174.653094] LustreError: 280721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756813401 with bad export cookie 18433659187229745461 [ 4174.656491] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4174.658027] LustreError: 280721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4174.803071] Lustre: server umount lustre-MDT0001 complete [ 4178.579137] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 4179.647223] Key type lgssc unregistered [ 4179.774277] LNet: 281918:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4179.776320] LNetError: 281918:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4179.787320] LNet: Removed LNI 192.168.204.115@tcp [ 4180.107317] Key type .llcrypt unregistered [ 4180.108335] Key type ._llcrypt unregistered [ 4191.546429] Key type ._llcrypt registered [ 4191.547370] Key type .llcrypt registered [ 4191.579639] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 4197.818024] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4198.155564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4198.194329] alg: No test for adler32 (adler32-zlib) [ 4199.060114] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 4199.163098] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 4200.744169] Key type lgssc registered [ 4201.064708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4203.874338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4206.048144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4207.971235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4209.941314] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4212.051968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4212.068351] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4212.073547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4213.153291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4213.165239] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4213.200562] Lustre: lustre-MDT0000: new disk, initializing [ 4213.223841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4213.230087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4214.512336] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4218.564686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4218.585869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4218.628678] Lustre: 285826:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4218.656941] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4218.659780] Lustre: Skipped 1 previous similar message [ 4218.703890] Lustre: lustre-MDT0001: new disk, initializing [ 4218.741957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4218.756985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4218.759817] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4220.022521] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4223.078595] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4224.326219] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4226.730314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4226.752015] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4226.849332] Lustre: lustre-OST0000: new disk, initializing [ 4226.851730] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4226.887601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4228.825463] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4231.930613] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4232.192489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4232.195737] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4232.236771] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4240.754066] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 4243.166116] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 4252.725878] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 4296.675613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4296.676293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4296.678641] Lustre: Skipped 1 previous similar message [ 4299.885073] LustreError: 289436:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4299.945057] Lustre: server umount lustre-OST0000 complete [ 4301.794113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4301.794768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4301.797316] Lustre: Skipped 1 previous similar message [ 4301.800425] Lustre: Skipped 2 previous similar messages [ 4306.913241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4306.915107] Lustre: Skipped 1 previous similar message [ 4307.261680] LustreError: 289637:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4307.263422] LustreError: 289637:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4307.289624] Lustre: server umount lustre-MDT0000 complete [ 4308.484284] LustreError: 289110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756813535 with bad export cookie 13495779622959138113 [ 4308.486486] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4308.487288] LustreError: 289110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4308.552912] LustreError: 289840:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4308.554782] LustreError: 289840:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4308.665029] Lustre: server umount lustre-MDT0001 complete [ 4311.512848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4312.905861] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4314.442175] LustreError: 290780:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4314.444246] LustreError: 290780:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4314.468614] Lustre: server umount MGS complete [ 4318.009330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4318.158563] LustreError: 291414:0:(qmt_pool.c:1513:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 4318.183170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4319.398629] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4322.084265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4322.192248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4323.393241] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4324.953511] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4326.074299] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4328.163000] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4330.004288] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4331.719384] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4367.328995] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4367.332274] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4367.338515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4368.354333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4372.654991] LustreError: 294080:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4372.658556] LustreError: 294080:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4372.725924] Lustre: server umount lustre-OST0000 complete [ 4378.593834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4378.593864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4378.594922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4378.594927] Lustre: Skipped 2 previous similar messages [ 4378.601715] Lustre: Skipped 2 previous similar messages [ 4388.320137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4388.349270] LustreError: 294281:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4388.350992] LustreError: 294281:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4388.377747] Lustre: server umount lustre-MDT0000 complete [ 4388.834092] LustreError: 291401:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4388.840969] LustreError: 291401:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 4389.555419] LustreError: 293816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756813616 with bad export cookie 13495779622959139926 [ 4389.559577] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4389.560698] LustreError: 293816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4389.696481] Lustre: server umount lustre-MDT0001 complete [ 4393.174951] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 4394.168660] Key type lgssc unregistered [ 4394.300310] LNet: 295017:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4394.302331] LNetError: 295017:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4394.309300] LNet: Removed LNI 192.168.204.115@tcp [ 4394.578826] Key type .llcrypt unregistered [ 4394.579808] Key type ._llcrypt unregistered [ 4400.532607] Key type ._llcrypt registered [ 4400.534519] Key type .llcrypt registered [ 4400.569955] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 4406.218862] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4406.493895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4406.499839] alg: No test for adler32 (adler32-zlib) [ 4407.344266] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 4407.424377] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 4409.000146] Key type lgssc registered [ 4409.329675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4411.923428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4413.703673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4416.281209] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4418.850273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4420.870477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4420.887379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4420.892443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4421.969488] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4421.979144] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4422.021608] Lustre: lustre-MDT0000: new disk, initializing [ 4422.047723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4422.054849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4423.203265] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4427.256574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4427.284756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4427.380488] Lustre: lustre-OST0000: new disk, initializing [ 4427.382292] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4427.383787] Lustre: Skipped 1 previous similar message [ 4427.407612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4429.347165] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4435.964420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 4435.967984] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 4435.975962] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 4604.897652] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4604.903925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4608.386972] LustreError: 299928:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 4608.502385] Lustre: server umount lustre-MDT0000 complete [ 4609.706213] LustreError: 298176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756813836 with bad export cookie 17985283655101705149 [ 4609.708427] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4609.710383] LustreError: 298176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4609.728811] LustreError: 300131:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4609.731987] LustreError: 300131:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4609.803620] Lustre: server umount lustre-OST0000 complete [ 4617.722376] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 4620.693472] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4624.066698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4626.267169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4628.220738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4630.308961] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4632.730995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4632.758467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4632.847543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4632.858474] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4632.898341] Lustre: lustre-MDT0000: new disk, initializing [ 4632.922203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4632.929910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4634.300763] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4638.391966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4638.415863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4638.448156] Lustre: 303166:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4638.474465] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4638.476057] Lustre: Skipped 1 previous similar message [ 4638.512432] Lustre: lustre-MDT0001: new disk, initializing [ 4638.548177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4638.564501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4638.568361] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4639.859351] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4642.836589] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4643.957993] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4646.165151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4646.187399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4646.311871] Lustre: lustre-OST0000: new disk, initializing [ 4646.314723] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4646.349041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4648.257353] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4651.271832] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4652.612505] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4653.560236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4653.563606] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4653.594214] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4653.708639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4654.963997] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4655.033409] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4658.658713] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4658.660103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4658.661179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4658.664282] Lustre: Skipped 1 previous similar message [ 4661.742791] LustreError: 306109:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4661.745482] LustreError: 306109:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4661.815079] Lustre: server umount lustre-OST0000 complete [ 4663.777527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4663.778120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4663.783290] Lustre: Skipped 1 previous similar message [ 4663.784906] Lustre: Skipped 1 previous similar message [ 4668.897698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4668.900903] Lustre: Skipped 2 previous similar messages [ 4669.183352] LustreError: 306310:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4669.185124] LustreError: 306310:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4669.212920] Lustre: server umount lustre-MDT0000 complete [ 4670.454424] LustreError: 303157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756813897 with bad export cookie 17985283655104645940 [ 4670.459759] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4670.463364] LustreError: 303157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4670.614203] Lustre: server umount lustre-MDT0001 complete [ 4671.749868] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [ 4672.332796] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 07:51:38 (1756813898) [ 4674.579185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4674.776610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4676.147733] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4678.853190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4680.309537] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4682.109457] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4683.266593] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4685.531848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4687.452873] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4689.098550] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4691.467802] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4691.493737] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4691.543325] Lustre: lustre-OST0001: new disk, initializing [ 4691.545161] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4691.573979] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4691.576051] Lustre: Skipped 2 previous similar messages [ 4693.434423] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4696.573362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4696.576457] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4696.591639] LustreError: 310139:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 4696.595611] LustreError: 310139:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 4697.635716] LustreError: 310139:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 4697.640562] LustreError: 310139:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 4698.659375] LustreError: 310139:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 4698.662395] LustreError: 310139:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 4700.715855] LustreError: 310144:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 4700.721908] LustreError: 310144:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 4702.319608] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4702.394604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4703.834572] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9750406d1800.ost_server_uuid 50 [ 4704.422810] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9750406d1800.ost_server_uuid in FULL state after 0 sec [ 4704.806691] LustreError: 310144:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 4704.813473] LustreError: 310144:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [ 4706.004724] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9750406d1800.ost_server_uuid 50 [ 4706.620782] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9750406d1800.ost_server_uuid in FULL state after 0 sec [ 4707.885329] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4719.016440] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4719.020713] Lustre: Skipped 1 previous similar message [ 4719.022163] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4719.026023] Lustre: Skipped 1 previous similar message [ 4724.659204] LustreError: 310883:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4724.664024] LustreError: 310883:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4724.697979] Lustre: server umount lustre-OST0001 complete [ 4729.318054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4729.319942] LustreError: 309188:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4729.321388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4729.321399] Lustre: Skipped 3 previous similar messages [ 4729.325917] Lustre: Skipped 1 previous similar message [ 4729.334920] LustreError: 309188:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 4732.785203] LustreError: 311085:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4732.790097] LustreError: 311085:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4732.900682] Lustre: server umount lustre-OST0000 complete [ 4735.457777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4735.462554] LustreError: Skipped 1 previous similar message [ 4739.556637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4739.564041] Lustre: Skipped 1 previous similar message [ 4749.280134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4749.319999] LustreError: 311286:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4749.322726] LustreError: 311286:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4749.356626] Lustre: server umount lustre-MDT0000 complete [ 4749.793347] LustreError: 307082:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4749.797724] LustreError: 307082:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 4750.706267] LustreError: 311287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756813977 with bad export cookie 17985283655104646822 [ 4750.709460] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4750.711716] LustreError: 311287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4750.866521] Lustre: server umount lustre-MDT0001 complete [ 4755.008213] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 4756.145364] Key type lgssc unregistered [ 4756.272427] LNet: 312020:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4756.275505] LNetError: 312020:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4756.283366] LNet: Removed LNI 192.168.204.115@tcp [ 4756.639527] Key type .llcrypt unregistered [ 4756.640674] Key type ._llcrypt unregistered [ 4766.557592] Key type ._llcrypt registered [ 4766.558934] Key type .llcrypt registered [ 4766.607415] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4766.977337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4767.027560] alg: No test for adler32 (adler32-zlib) [ 4767.884758] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 4767.984052] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 4769.568119] Key type lgssc registered [ 4769.917350] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4773.037139] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4773.042786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4774.226611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4775.660459] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4777.977843] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4780.646984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4780.767469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4782.287912] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4786.122248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4786.281391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4788.169192] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4789.950037] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4790.380312] LustreError: 314951:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4790.390429] LustreError: 314951:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 4792.758968] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4792.863114] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4794.943341] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4796.767653] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4799.983049] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 4799.983125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4803.853541] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4803.932223] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4881.895431] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4881.897409] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4881.904306] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4881.905719] Lustre: Skipped 1 previous similar message [ 4887.011510] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4887.013159] Lustre: Skipped 1 previous similar message [ 4887.084298] LustreError: 316818:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4887.098861] Lustre: server umount lustre-OST0001 complete [ 4892.132442] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4892.133542] LustreError: 316396:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4892.136233] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4892.136579] Lustre: Skipped 1 previous similar message [ 4892.140569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4892.147669] LustreError: 316396:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 4892.151220] Lustre: Skipped 1 previous similar message [ 4894.636653] LustreError: 317020:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4894.638347] LustreError: 317020:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4894.705801] Lustre: server umount lustre-OST0000 complete [ 4897.251204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4897.252177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4897.256020] Lustre: Skipped 2 previous similar messages [ 4897.258974] Lustre: Skipped 1 previous similar message [ 4902.339091] LustreError: 317223:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4902.341925] LustreError: 317223:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4902.366594] Lustre: server umount lustre-MDT0000 complete [ 4902.369376] LustreError: 314224:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4902.373505] LustreError: 314224:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 4903.558707] LustreError: 313378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756814130 with bad export cookie 10076030909707076497 [ 4903.561232] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4903.562113] LustreError: 313378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4903.729063] Lustre: server umount lustre-MDT0001 complete [ 4907.491663] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 4908.645266] Key type lgssc unregistered [ 4908.784784] LNet: 317957:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4908.787180] LNetError: 317957:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4908.795507] LNet: Removed LNI 192.168.204.115@tcp [ 4909.126868] Key type .llcrypt unregistered [ 4909.128291] Key type ._llcrypt unregistered [ 4926.551238] Key type ._llcrypt registered [ 4926.552446] Key type .llcrypt registered [ 4926.587239] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4926.865316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4926.875106] alg: No test for adler32 (adler32-zlib) [ 4927.725500] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 4927.810847] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 4929.384155] Key type lgssc registered [ 4929.766442] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4932.740180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4932.746906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4933.894710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4935.352742] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4938.335892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4938.494483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4939.826216] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4940.877870] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4943.173069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4943.321043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4944.365907] LustreError: 321819:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4944.370999] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4944.376643] LustreError: 321819:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 4945.516031] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4948.453846] LustreError: 321664:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4948.464038] LustreError: 321664:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 4948.789883] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4948.926342] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4951.311843] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4951.734642] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 4955.882914] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4957.463802] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4969.446135] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4969.448262] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4969.450437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4969.462761] Lustre: Skipped 3 previous similar messages [ 4973.059748] LustreError: 323613:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4973.091326] Lustre: server umount lustre-MDT0000 complete [ 4974.561772] LustreError: 320203:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4974.567169] LustreError: 320203:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 4974.629457] LustreError: 321312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756814201 with bad export cookie 14650274848656516300 [ 4974.634800] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4974.635420] LustreError: 321312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4974.709489] LustreError: 323815:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 4974.711309] LustreError: 323815:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4974.780844] Lustre: server umount lustre-MDT0001 complete [ 4986.870077] LustreError: 324016:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4986.872546] LustreError: 324016:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4986.886890] Lustre: server umount lustre-OST0000 complete [ 4998.648703] LustreError: 324218:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4998.652622] LustreError: 324218:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4998.761755] Lustre: server umount lustre-OST0001 complete [ 5005.143363] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5009.539852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5009.768980] LustreError: 325396:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5009.801433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5011.317209] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5014.706680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5016.575602] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5018.002722] Lustre: 326505:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5021.277328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5021.433649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5021.436395] Lustre: Skipped 1 previous similar message [ 5024.031279] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5024.492567] LustreError: 326863:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5027.917151] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5030.376972] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5034.676312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5034.684377] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 5039.794826] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5041.377692] Lustre: 328355:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5050.346180] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5050.348767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5050.354827] Lustre: Skipped 3 previous similar messages [ 5050.358453] Lustre: Skipped 5 previous similar messages [ 5053.905380] LustreError: 328459:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5053.909284] LustreError: 328459:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5053.949564] Lustre: server umount lustre-MDT0000 complete [ 5055.461685] LustreError: 325395:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5055.472346] LustreError: 325395:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 5055.722922] LustreError: 326509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756814282 with bad export cookie 14650274848656517959 [ 5055.727402] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5055.730043] LustreError: 326509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5055.915554] Lustre: server umount lustre-MDT0001 complete [ 5067.769163] LustreError: 328861:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 5067.772587] LustreError: 328861:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 5067.796264] Lustre: server umount lustre-OST0000 complete [ 5079.799736] Lustre: server umount lustre-OST0001 complete [ 5081.056500] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [ 5081.717713] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [ 5082.467821] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 07:58:28 (1756814308) [ 5091.595963] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5095.153791] loop0: detected capacity change from 0 to 1099511627776 [ 5095.344895] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 5095.347878] *** VALIDATE xfs *** [ 5095.352626] XFS (loop0): Mounting V5 Filesystem [ 5095.362156] XFS (loop0): Ending clean mount [ 5095.905778] loop1: detected capacity change from 0 to 18691697672192 [ 5111.079375] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5118.425473] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 5121.969914] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5126.082278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5129.404296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5131.913141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5135.074981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5138.289949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5138.328217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5138.432949] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5138.452186] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5138.515206] Lustre: lustre-MDT0000: new disk, initializing [ 5138.560865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5138.564822] Lustre: Skipped 1 previous similar message [ 5138.575406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5140.192453] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5145.468299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5145.514252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5145.577543] Lustre: 333430:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5145.619599] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5145.622246] Lustre: Skipped 1 previous similar message [ 5145.660991] Lustre: lustre-MDT0001: new disk, initializing [ 5145.740818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5145.746806] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5147.510476] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5150.759034] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5151.898463] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5154.065024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5154.092024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5154.190407] Lustre: lustre-OST0000: new disk, initializing [ 5154.192280] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5155.449981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5155.452934] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5155.488064] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5155.984654] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5158.940948] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5160.223481] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5160.302267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5161.648308] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5161.719277] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5164.398364] LustreError: 336368:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5164.400165] LustreError: 336368:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5164.467332] Lustre: server umount lustre-OST0000 complete [ 5166.050473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5166.050530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5166.054794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5166.057579] Lustre: Skipped 2 previous similar messages [ 5171.169280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5171.171183] Lustre: Skipped 1 previous similar message [ 5175.286889] Lustre: server umount lustre-MDT0000 complete [ 5176.289089] LustreError: 333441:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5176.295412] LustreError: 333441:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 5176.436201] LustreError: 333422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756814403 with bad export cookie 14650274848656519513 [ 5176.439072] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5176.441769] LustreError: 333422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5187.153638] Lustre: server umount lustre-MDT0001 complete [ 5190.282715] XFS (loop0): Unmounting Filesystem [ 5192.000167] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 08:00:18 (1756814418) [ 5194.591683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5194.764546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5194.766432] Lustre: Skipped 2 previous similar messages [ 5196.003886] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5199.690024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5201.069776] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5203.249472] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5204.452624] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5206.715938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5208.929830] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5210.558177] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5225.444095] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5225.444725] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5225.446406] LustreError: Skipped 1 previous similar message [ 5225.450941] Lustre: Skipped 1 previous similar message [ 5225.453375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5225.455561] Lustre: Skipped 1 previous similar message [ 5226.484121] LustreError: 339956:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5226.488159] LustreError: 339956:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5226.585831] Lustre: server umount lustre-OST0000 complete [ 5230.421095] Lustre: server umount lustre-MDT0000 complete [ 5230.562007] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5230.562066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5230.565718] LustreError: 337587:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5230.570641] LustreError: 337587:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 5231.726230] LustreError: 337567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756814458 with bad export cookie 14650274848656520395 [ 5231.729433] LustreError: 337567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5231.729611] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5231.872583] Lustre: server umount lustre-MDT0001 complete [ 5235.432355] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 5236.450171] Key type lgssc unregistered [ 5236.607537] LNet: 340900:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5236.610304] LNetError: 340900:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5236.618401] LNet: Removed LNI 192.168.204.115@tcp [ 5236.980145] Key type .llcrypt unregistered [ 5236.981949] Key type ._llcrypt unregistered [ 5247.595201] Key type ._llcrypt registered [ 5247.596995] Key type .llcrypt registered [ 5247.655687] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5248.018117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5248.050667] alg: No test for adler32 (adler32-zlib) [ 5248.899080] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 5248.979989] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 5250.560113] Key type lgssc registered [ 5250.947521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5254.199662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5254.206130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5255.355903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5256.566908] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5260.177907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5260.380992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5262.153684] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5264.737146] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5266.448520] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5269.217306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5269.396575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5271.323334] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5273.044543] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5286.883567] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5286.889836] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5286.891728] Lustre: Skipped 1 previous similar message [ 5288.717333] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.15@tcp (stopping) [ 5292.001703] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5292.092063] LustreError: 344742:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5292.111572] Lustre: server umount lustre-MDT0001 complete [ 5293.006934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5293.830426] LustreError: 342421:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5293.835179] LustreError: 342421:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 5295.009830] LustreError: 342425:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5295.013730] LustreError: 342425:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 5295.370397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5295.397639] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5295.399764] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5295.413262] Lustre: 342414:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5295.519657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5296.669426] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5300.714068] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5304.293585] LustreError: 345920:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 5304.296060] LustreError: 345920:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 5304.298676] LustreError: 342417:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 5304.314678] LustreError: 345923:0:(obd_config.c:1996:class_config_llog_handler()) MGC192.168.204.115@tcp: cfg command failed: rc = -17 [ 5304.319725] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 5304.319725] [ 5601.036291] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.15@tcp (stopping) [ 5601.038140] Lustre: Skipped 1 previous similar message [ 5602.786684] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5602.787329] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5602.790624] Lustre: Skipped 2 previous similar messages [ 5605.434243] LustreError: 357161:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5605.436314] LustreError: 357161:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5605.459027] Lustre: server umount lustre-MDT0001 complete [ 5606.150183] LustreError: 344592:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5606.370450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5608.749909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5608.782269] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5608.785285] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5608.800070] Lustre: 342414:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5608.907857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5610.076164] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5614.054635] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5617.639388] LustreError: 358382:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 5617.641492] LustreError: 358382:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 5617.643722] LustreError: 342417:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 5617.646909] LustreError: 342417:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 5617.666735] LustreError: 358389:0:(obd_config.c:688:class_setup()) Device 8 already setup (type osp) [ 5617.668608] LustreError: 358389:0:(obd_config.c:1996:class_config_llog_handler()) MGC192.168.204.115@tcp: cfg command failed: rc = -17 [ 5617.670908] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.204.115@tcp [ 5617.670908] [ 5913.363709] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.15@tcp (stopping) [ 5913.368413] Lustre: Skipped 2 previous similar messages [ 5916.134507] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5916.134849] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5916.146357] Lustre: Skipped 1 previous similar message [ 5918.799738] LustreError: 369474:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5918.804507] LustreError: 369474:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5918.839829] Lustre: server umount lustre-MDT0001 complete [ 5920.062655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5921.252970] LustreError: 342425:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5921.263163] LustreError: 342425:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 5923.084986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5923.120335] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5923.123576] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5923.141334] Lustre: 342412:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5923.265206] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 5924.686214] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5928.423054] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5929.958159] LustreError: 370567:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 5929.960571] LustreError: 370567:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 5929.962852] LustreError: 342417:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 5929.964704] LustreError: 342417:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 5929.982483] LustreError: 370576:0:(lod_lov.c:207:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 5929.986903] LustreError: 370576:0:(obd_config.c:1996:class_config_llog_handler()) MGC192.168.204.115@tcp: cfg command failed: rc = -114 [ 5929.989235] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 5929.989235] [ 6240.740147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6240.741613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6240.758101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6240.762127] Lustre: Skipped 5 previous similar messages [ 6244.813615] LustreError: 381809:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6244.818078] LustreError: 381809:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6244.854957] Lustre: server umount lustre-MDT0000 complete [ 6245.858209] LustreError: 344592:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6245.866466] LustreError: 344592:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 6246.396273] LustreError: 344744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756815473 with bad export cookie 14442547456142707738 [ 6246.402793] LustreError: 344744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6246.403063] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6250.978782] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6250.986245] Lustre: Skipped 2 previous similar messages [ 6252.550876] LustreError: 382010:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6252.554691] LustreError: 382010:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6252.634992] Lustre: server umount lustre-MDT0001 complete [ 6254.285200] Lustre: server umount lustre-OST0000 complete [ 6257.568413] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 6260.447842] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6263.684494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6265.803334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6267.796486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6269.826621] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6272.333407] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 08:18:18 (1756815498) [ 6275.355770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6275.391914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6275.506679] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6275.526512] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6275.576355] Lustre: lustre-MDT0000: new disk, initializing [ 6275.605618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6275.614416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6277.019500] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6281.227109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6281.251794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6281.285431] Lustre: 384742:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6281.309115] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6281.311139] Lustre: Skipped 1 previous similar message [ 6281.345458] Lustre: lustre-MDT0001: new disk, initializing [ 6281.378643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6281.395280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6281.400305] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6282.668124] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6285.692192] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6287.007907] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6289.345374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6289.385604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6289.519434] Lustre: lustre-OST0000: new disk, initializing [ 6289.522399] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6289.567809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6291.733593] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6292.795939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6292.798893] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6292.832226] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6294.996264] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6298.083271] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6298.084290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6298.090289] Lustre: Skipped 2 previous similar messages [ 6298.092488] Lustre: Skipped 3 previous similar messages [ 6303.696691] LustreError: 387181:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6303.700442] LustreError: 387181:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6303.738173] Lustre: server umount lustre-MDT0000 complete [ 6305.289805] LustreError: 385522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756815532 with bad export cookie 14442547456142709698 [ 6305.294596] LustreError: 385534:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [ 6305.296515] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6305.458992] Lustre: server umount lustre-MDT0001 complete [ 6317.554605] Lustre: server umount lustre-OST0000 complete [ 6327.660804] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 6330.721367] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6334.428058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6337.393509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6339.532144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6341.858787] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6342.414981] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6136 sec ========== 08:19:28 (1756815568) [ 6342.991616] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:19:29 (1756815569) === [ 6344.316881] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:19:30 (1756815570) === [ 6358.570099] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6360.023466] Key type lgssc unregistered [ 6360.169359] LNet: 392685:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6360.172539] LNetError: 392685:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6360.180348] LNet: Removed LNI 192.168.204.115@tcp [ 6360.541307] Key type .llcrypt unregistered [ 6360.543799] Key type ._llcrypt unregistered