[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 466546964 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002310] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004020] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010140] LSM: Security Framework initializing [ 0.012030] Yama: becoming mindful. [ 0.013027] SELinux: Initializing. [ 0.013828] *** VALIDATE selinux *** [ 0.021413] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026222] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028153] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030114] *** VALIDATE tmpfs *** [ 0.032096] *** VALIDATE proc *** [ 0.033261] *** VALIDATE cgroup *** [ 0.034012] *** VALIDATE cgroup2 *** [ 0.035291] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036164] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039038] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.042871] debug: unmapping init [mem 0xffffffff87a59000-0xffffffff87a60fff] [ 0.044159] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045776] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046030] ... version: 2 [ 0.047015] ... bit width: 48 [ 0.048015] ... generic registers: 4 [ 0.049021] ... value mask: 0000ffffffffffff [ 0.050017] ... max period: 00007fffffffffff [ 0.051013] ... fixed-purpose events: 3 [ 0.052024] ... event mask: 000000070000000f [ 0.053313] rcu: Hierarchical SRCU implementation. [ 0.055527] smp: Bringing up secondary CPUs ... [ 0.056561] x86: Booting SMP configuration: [ 0.057029] .... node #0, CPUs: #1 #2 #3 [ 0.061048] smp: Brought up 1 node, 4 CPUs [ 0.063016] smpboot: Max logical packages: 1 [ 0.064014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.129716] node 0 deferred pages initialised in 63ms [ 0.132246] devtmpfs: initialized [ 0.133230] x86/mm: Memory block size: 128MB [ 0.135230] gcov: version magic: 0x41383552 [ 0.137276] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.138097] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.139293] pinctrl core: initialized pinctrl subsystem [ 0.140158] [ 0.140715] ************************************************************* [ 0.141026] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.142012] ** ** [ 0.143015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.144013] ** ** [ 0.145012] ** This means that this kernel is built to expose internal ** [ 0.146016] ** IOMMU data structures, which may compromise security on ** [ 0.147013] ** your system. ** [ 0.148012] ** ** [ 0.149015] ** If you see this message and you are not debugging the ** [ 0.150013] ** kernel, report this immediately to your vendor! ** [ 0.151012] ** ** [ 0.152016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153013] ************************************************************* [ 0.154605] NET: Registered protocol family 16 [ 0.155319] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.156053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.157052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.158421] cpuidle: using governor menu [ 0.160823] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.164696] PCI: Using configuration type 1 for base access [ 0.167160] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.176140] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.177038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.179052] cryptd: max_cpu_qlen set to 1000 [ 0.181042] ACPI: Added _OSI(Module Device) [ 0.182012] ACPI: Added _OSI(Processor Device) [ 0.183020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.184015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.187985] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.190741] ACPI: Interpreter enabled [ 0.191069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.192016] ACPI: Using IOAPIC for interrupt routing [ 0.193108] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.194422] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.204012] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.205072] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.206024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.207086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.209617] acpiphp: Slot [2] registered [ 0.210131] acpiphp: Slot [5] registered [ 0.211121] acpiphp: Slot [6] registered [ 0.212148] acpiphp: Slot [7] registered [ 0.213136] acpiphp: Slot [8] registered [ 0.214120] acpiphp: Slot [9] registered [ 0.215173] acpiphp: Slot [10] registered [ 0.216122] acpiphp: Slot [3] registered [ 0.217112] acpiphp: Slot [4] registered [ 0.218133] acpiphp: Slot [11] registered [ 0.219077] acpiphp: Slot [12] registered [ 0.220042] acpiphp: Slot [13] registered [ 0.221061] acpiphp: Slot [14] registered [ 0.222069] acpiphp: Slot [15] registered [ 0.223092] acpiphp: Slot [16] registered [ 0.225118] acpiphp: Slot [17] registered [ 0.227092] acpiphp: Slot [18] registered [ 0.228108] acpiphp: Slot [19] registered [ 0.230108] acpiphp: Slot [20] registered [ 0.232114] acpiphp: Slot [21] registered [ 0.234112] acpiphp: Slot [22] registered [ 0.235126] acpiphp: Slot [23] registered [ 0.237109] acpiphp: Slot [24] registered [ 0.239119] acpiphp: Slot [25] registered [ 0.240100] acpiphp: Slot [26] registered [ 0.242116] acpiphp: Slot [27] registered [ 0.244105] acpiphp: Slot [28] registered [ 0.245100] acpiphp: Slot [29] registered [ 0.247137] acpiphp: Slot [30] registered [ 0.248164] acpiphp: Slot [31] registered [ 0.250103] PCI host bridge to bus 0000:00 [ 0.252042] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.254035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.257037] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.260037] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.262031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.264037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.266323] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.270069] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.273411] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.281028] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.285067] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.288028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.290022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.292028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.294620] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.299164] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.302035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.303643] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.308012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.321015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.324896] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.329786] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.337025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.344028] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.370040] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.381753] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.386016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.392018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.413015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.421055] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.431000] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.435018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.457016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.464385] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.469013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.474016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.488017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.496136] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.501014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.506017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.518023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.526290] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.531017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.536021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.553023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.565109] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.567771] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.571560] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.574356] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.575322] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.582111] iommu: Default domain type: Passthrough [ 0.585460] SCSI subsystem initialized [ 0.587378] ACPI: bus type USB registered [ 0.588162] usbcore: registered new interface driver usbfs [ 0.590187] usbcore: registered new interface driver hub [ 0.593114] usbcore: registered new device driver usb [ 0.595387] pps_core: LinuxPPS API ver. 1 registered [ 0.598014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.602272] PTP clock support registered [ 0.605112] EDAC MC: Ver: 3.0.0 [ 0.606000] PCI: Using ACPI for IRQ routing [ 0.609426] NetLabel: Initializing [ 0.610011] NetLabel: domain hash size = 128 [ 0.612013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.615085] NetLabel: unlabeled traffic allowed by default [ 0.617271] vgaarb: loaded [ 0.619410] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.622064] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.631639] clocksource: Switched to clocksource kvm-clock [ 0.832382] VFS: Disk quotas dquot_6.6.0 [ 0.834556] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.838825] *** VALIDATE ramfs *** [ 0.842992] *** VALIDATE hugetlbfs *** [ 0.847938] pnp: PnP ACPI init [ 0.859462] pnp: PnP ACPI: found 6 devices [ 0.919806] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.926462] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.929835] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.932864] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.937122] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.945573] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.958526] NET: Registered protocol family 2 [ 0.962450] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.968884] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.974259] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.981391] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.985828] TCP: Hash tables configured (established 65536 bind 65536) [ 0.989843] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.993971] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.998327] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.008324] NET: Registered protocol family 1 [ 1.016896] RPC: Registered named UNIX socket transport module. [ 1.023944] RPC: Registered udp transport module. [ 1.028875] RPC: Registered tcp transport module. [ 1.035622] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.039387] NET: Registered protocol family 44 [ 1.041138] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.049884] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.052077] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.064915] pci 0000:00:01.0: quirk_isa_dma_hangs+0x0/0x20 took 12537 usecs [ 1.073636] PCI: CLS 0 bytes, default 64 [ 1.080881] Unpacking initramfs... [ 4.170200] debug: unmapping init [mem 0xffff88d77cc54000-0xffff88d77ffbffff] [ 4.184486] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.189152] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.195855] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.429715] Initialise system trusted keyrings [ 5.430934] Key type blacklist registered [ 5.435406] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.459475] zbud: loaded [ 5.475799] *** VALIDATE nfs *** [ 5.479856] *** VALIDATE nfs4 *** [ 5.481363] pstore: using deflate compression [ 5.506270] Platform Keyring initialized [ 5.818865] NET: Registered protocol family 38 [ 5.823794] Key type asymmetric registered [ 5.827805] Asymmetric key parser 'x509' registered [ 5.835492] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.844092] io scheduler mq-deadline registered [ 5.849971] io scheduler kyber registered [ 5.853913] io scheduler bfq registered [ 5.857750] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.864367] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.870119] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.877598] ACPI: Power Button [PWRF] [ 5.894993] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.917232] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.950073] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.967189] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.998815] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.048856] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.099396] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.119388] Non-volatile memory driver v1.3 [ 6.122045] Linux agpgart interface v0.103 [ 6.222522] virtio_blk virtio1: [vda] 149776 512-byte logical blocks (76.7 MB/73.1 MiB) [ 6.231145] vda: detected capacity change from 0 to 76685312 [ 6.284255] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.299624] vdb: detected capacity change from 0 to 1073741824 [ 6.336034] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.338984] vdc: detected capacity change from 0 to 2621440000 [ 6.388608] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.395235] vdd: detected capacity change from 0 to 2621440000 [ 6.434747] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.439702] vde: detected capacity change from 0 to 4294967296 [ 6.472834] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.480063] vdf: detected capacity change from 0 to 4294967296 [ 6.508986] libphy: Fixed MDIO Bus: probed [ 6.537676] usbcore: registered new interface driver usbserial_generic [ 6.540282] usbserial: USB Serial support registered for generic [ 6.542839] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.547678] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.549372] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.552029] mousedev: PS/2 mouse device common for all mice [ 6.555777] rtc_cmos 00:05: RTC can wake from S4 [ 6.558174] rtc_cmos 00:05: registered as rtc0 [ 6.563488] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.573383] intel_pstate: CPU model not supported [ 6.576458] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.582081] hid: raw HID events driver (C) Jiri Kosina [ 6.591623] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.592832] usbcore: registered new interface driver usbhid [ 6.592840] usbhid: USB HID core driver [ 6.592966] drop_monitor: Initializing network drop monitor service [ 6.593154] Initializing XFRM netlink socket [ 6.593981] NET: Registered protocol family 10 [ 6.619236] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.619632] Segment Routing with IPv6 [ 6.629146] NET: Registered protocol family 17 [ 6.635283] mpls_gso: MPLS GSO support [ 6.646560] RAS: Correctable Errors collector initialized. [ 6.650776] AVX version of gcm_enc/dec engaged. [ 6.654835] AES CTR mode by8 optimization enabled [ 6.789118] sched_clock: Marking stable (6789082537, 0)->(7774466946, -985384409) [ 6.799996] registered taskstats version 1 [ 6.810905] Loading compiled-in X.509 certificates [ 6.814521] zswap: loaded using pool lzo/zbud [ 6.865473] Key type big_key registered [ 6.895874] Key type encrypted registered [ 6.898001] ima: No TPM chip found, activating TPM-bypass! [ 6.902926] ima: Allocated hash algorithm: sha1 [ 6.905980] ima: No architecture policies found [ 6.910759] evm: Initialising EVM extended attributes: [ 6.915754] evm: security.selinux [ 6.917810] evm: security.ima [ 6.921657] evm: security.capability [ 6.925563] evm: HMAC attrs: 0x1 [ 6.930380] rtc_cmos 00:05: setting system clock to 2026-09-05 05:28:58 UTC (1788586138) [ 6.945985] debug: unmapping init [mem 0xffffffff88a03000-0xffffffff88bfffff] [ 6.955296] debug: unmapping init [mem 0xffffffff87782000-0xffffffff87a58fff] [ 6.968395] Write protecting the kernel read-only data: 28672k [ 6.974588] debug: unmapping init [mem 0xffffffff85e03000-0xffffffff85ffffff] [ 6.980854] debug: unmapping init [mem 0xffffffff86714000-0xffffffff867fffff] [ 7.074730] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.098064] systemd[1]: Detected virtualization kvm. [ 7.101073] systemd[1]: Detected architecture x86-64. [ 7.104684] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.165922] systemd[1]: No hostname configured. [ 7.168782] systemd[1]: Set hostname to . [ 7.172413] random: systemd: uninitialized urandom read (16 bytes read) [ 7.180913] systemd[1]: Initializing machine ID from random generator. [ 7.578383] random: systemd: uninitialized urandom read (16 bytes read) [ 7.581256] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 7.592390] random: systemd: uninitialized urandom read (16 bytes read) [ 7.600397] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 7.620653] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.488935] device-mapper: uevent: version 1.0.3 [ 9.492553] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 11.131456] random: fast init done [ 11.350743] virtio_net virtio0 ens2: renamed from eth0 [ 11.599157] scsi host0: ata_piix [ 11.718860] scsi host1: ata_piix [ 11.723080] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.730908] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.438752] random: crng init done [ 16.445700] random: 7 urandom warning(s) missed due to ratelimiting [ 18.501887] dracut-initqueue[579]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 20.422847] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. [ 21.134375] hrtimer: interrupt took 2922772 ns Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.091615] printk: systemd: 26 output lines suppressed due to ratelimiting [ 24.005548] SELinux: Disabled at runtime. [ 24.164327] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.188078] systemd[1]: Detected virtualization kvm. [ 24.194488] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.386768] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.396109] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.409981] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.422608] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.428139] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.440658] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.450428] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK [0[ 25.626611] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS m] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 27.106301] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 28.046019] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.130231] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.516721] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.629910] EDAC sbridge: Ver: 1.1.2 [ 32.387793] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 33.288841] NFS: Registering the id_resolver key type [ 33.292162] Key type id_resolver registered [ 33.294103] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg259-server login: [ 89.657863] libcfs: loading out-of-tree module taints kernel. [ 89.708710] Key type ._llcrypt registered [ 89.711500] Key type .llcrypt registered [ 89.840970] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 108.030203] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 109.256807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 109.270263] alg: No test for adler32 (adler32-zlib) [ 110.916697] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 111.955431] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 113.791256] Key type lgssc registered [ 116.719441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 137.806359] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 184.646823] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 199.574398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 199.610802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 200.867997] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 200.896285] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 200.979512] Lustre: lustre-MDT0000: new disk, initializing [ 201.051410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 201.063654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 206.315085] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.896994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 221.993255] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 222.039451] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 222.045770] Lustre: Skipped 1 previous similar message [ 222.144370] Lustre: lustre-MDT0001: new disk, initializing [ 222.241063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 222.291402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 222.300855] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 227.845933] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 233.183549] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 244.743923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 245.019440] Lustre: lustre-OST0000: new disk, initializing [ 245.023613] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 245.035049] Lustre: 8453:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 245.105719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 252.650749] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 252.995546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 253.007114] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 253.138637] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 269.205500] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 269.402706] Lustre: lustre-OST0001: new disk, initializing [ 269.410268] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 269.421710] Lustre: 9525:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 269.546742] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 275.544194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 275.558974] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 275.665268] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 276.848948] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 289.230444] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 297.310734] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 304.494994] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing check_logdir /tmp/testlogs/ [ 310.163136] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing yml_node [ 314.882289] Lustre: DEBUG MARKER: Client: 2.17.58.3 [ 317.640769] Lustre: DEBUG MARKER: MDS: 2.17.58.3 [ 320.655811] Lustre: DEBUG MARKER: OSS: 2.17.58.3 [ 322.765243] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 01:34:12 EDT 2026 [ 341.126755] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 342.881131] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 360.416192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 360.430661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.464373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 362.977492] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 362.978104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 363.014779] Lustre: Skipped 3 previous similar messages [ 365.639985] Lustre: server umount lustre-MDT0000 complete [ 369.644162] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586501 with bad export cookie 14948445773376341520 [ 369.677627] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 373.221286] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 373.222889] LustreError: 10317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 373.223065] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 373.239304] Lustre: Skipped 3 previous similar messages [ 373.267329] LustreError: 10317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 374.244327] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 374.276985] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 375.946116] Lustre: server umount lustre-MDT0001 complete [ 380.650932] Lustre: server umount lustre-OST0000 complete [ 385.436860] Lustre: server umount lustre-OST0001 complete [ 392.996046] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 406.214799] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 451.945320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 452.328637] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 452.396763] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 452.576446] Lustre: lustre-MDT0000: new disk, initializing [ 452.717717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 452.740331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 458.818319] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 470.979772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 471.198176] Lustre: 15081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 471.335855] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 471.350691] Lustre: Skipped 1 previous similar message [ 471.505458] Lustre: lustre-MDT0001: new disk, initializing [ 471.716235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 471.807087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 471.814881] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 476.964755] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 487.690195] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 494.378778] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 502.216607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 502.549145] Lustre: lustre-OST0000: new disk, initializing [ 502.552818] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 502.564610] Lustre: 17048:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 502.726872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 504.272220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 504.292076] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 504.409134] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 511.284675] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 522.373861] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 527.597817] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 527.805938] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 532.743384] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 532.938721] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 540.131996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 540.133095] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 540.192854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 540.214122] LustreError: Skipped 1 previous similar message [ 540.249970] Lustre: Skipped 4 previous similar messages [ 544.301481] Lustre: server umount lustre-OST0000 complete [ 550.369470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 550.372390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 550.389363] Lustre: Skipped 2 previous similar messages [ 559.591518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 559.597411] Lustre: Skipped 4 previous similar messages [ 563.167280] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 563.584938] Lustre: server umount lustre-MDT0000 complete [ 564.705559] LustreError: 17056:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 564.728739] LustreError: 17056:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 568.199723] LustreError: 17060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586699 with bad export cookie 14948445773376343060 [ 568.209125] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 568.216900] LustreError: 17060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 568.828704] Lustre: server umount lustre-MDT0001 complete [ 574.509524] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 01:38:24 (1788586704) [ 581.587662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 582.147673] LustreError: 19318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 582.243369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 586.851199] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 587.237674] LustreError: 19319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.921756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 600.250910] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 608.556817] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 614.515542] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 620.764455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 627.092624] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 634.532296] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 641.114687] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 650.209270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 650.220861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 650.227963] Lustre: Skipped 2 previous similar messages [ 650.244143] Lustre: Skipped 2 previous similar messages [ 653.656490] Lustre: server umount lustre-MDT0000 complete [ 655.332151] LustreError: 19318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 655.345249] LustreError: 19318:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 657.354063] LustreError: 19299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586788 with bad export cookie 14948445773376343935 [ 657.371525] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 657.857427] Lustre: server umount lustre-MDT0001 complete [ 672.538935] Lustre: server umount lustre-OST0000 complete [ 695.761717] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 706.963909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 707.460926] LustreError: 23701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 707.543138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 707.549505] Lustre: Skipped 2 previous similar messages [ 712.491044] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 722.002838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 727.617144] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 731.187927] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 738.947765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 746.412680] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 755.615408] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 755.804681] Lustre: lustre-OST0001: new disk, initializing [ 755.808335] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 755.814190] Lustre: 26017:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 757.972180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 757.989213] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 758.105790] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 762.864992] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 771.897819] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 775.196741] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 788.963995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 788.979529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 788.980171] Lustre: Skipped 1 previous similar message [ 788.997232] Lustre: Skipped 3 previous similar messages [ 794.168474] Lustre: server umount lustre-MDT0000 complete [ 798.556181] LustreError: 26016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586930 with bad export cookie 14948445773376344789 [ 798.562740] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 798.566655] LustreError: 26016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 798.900655] Lustre: server umount lustre-MDT0001 complete [ 813.523295] Lustre: server umount lustre-OST0000 complete [ 814.559309] Lustre: 3649:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788586930/real 1788586930] req@ffff88d6c7b13480 x1875468608592768/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788586946 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 817.645484] Lustre: server umount lustre-OST0001 complete [ 834.364749] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 845.197946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 845.732291] LustreError: 29037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 845.765175] LustreError: 29037:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 845.836740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 845.852901] Lustre: Skipped 3 previous similar messages [ 850.843442] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 860.587159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 865.622650] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 868.312861] Lustre: 30187:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 874.879745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 882.060967] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 891.358299] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 898.756654] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 907.495372] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 912.197436] Lustre: 32066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 922.604499] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 922.609888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 922.612770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 922.612784] Lustre: Skipped 4 previous similar messages [ 922.619168] Lustre: Skipped 4 previous similar messages [ 924.655976] Lustre: server umount lustre-MDT0000 complete [ 927.718723] LustreError: 29034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 927.745932] LustreError: 29034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 928.544014] LustreError: 29017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587060 with bad export cookie 14948445773376346084 [ 928.555850] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 928.556822] LustreError: 29017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 928.856670] Lustre: server umount lustre-MDT0001 complete [ 943.256656] Lustre: server umount lustre-OST0000 complete [ 958.228660] Lustre: server umount lustre-OST0001 complete [ 967.319731] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 01:44:57 (1788587097) [ 995.920636] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1007.861244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1014.254180] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1024.418971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1029.549192] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1032.489852] Lustre: 36019:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1040.302102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1047.920414] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1057.451103] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1063.886935] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1072.841413] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1077.578796] Lustre: 37900:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1083.369776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1083.394126] Lustre: Skipped 2 previous similar messages [ 1083.405069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1083.410545] Lustre: Skipped 3 previous similar messages [ 1089.714076] Lustre: server umount lustre-MDT0000 complete [ 1093.605235] LustreError: 35895:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1093.620736] LustreError: 35895:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 1093.802220] LustreError: 34850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587225 with bad export cookie 14948445773376347183 [ 1093.803047] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1093.820468] LustreError: 34850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1094.423330] Lustre: server umount lustre-MDT0001 complete [ 1109.673346] Lustre: server umount lustre-OST0000 complete [ 1114.591195] Lustre: 3648:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788587230/real 1788587230] req@ffff88d7ff89a680 x1875468608694016/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788587246 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1115.392243] Lustre: server umount lustre-OST0001 complete [ 1127.603110] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 01:47:37 (1788587257) [ 1143.195670] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1152.989311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1153.453538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1153.466037] Lustre: Skipped 7 previous similar messages [ 1158.474177] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1169.079723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1175.272777] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1178.954661] Lustre: 41193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1185.682701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1192.929982] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1201.954775] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1209.343858] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1217.254870] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1220.873681] Lustre: 43070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1221.788877] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1221.793194] Lustre: Skipped 1 previous similar message [ 1225.700378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1225.716279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1225.726063] Lustre: Skipped 2 previous similar messages [ 1230.678846] Lustre: server umount lustre-MDT0000 complete [ 1234.680085] LustreError: 40022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587366 with bad export cookie 14948445773376348296 [ 1234.682614] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1234.685619] LustreError: 40022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1241.133813] Lustre: server umount lustre-MDT0001 complete [ 1249.522595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1259.999886] LustreError: 3645:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff88d6c7b10700 x1875468608746752/t0(0) o250->MGC192.168.202.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1265.094408] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1269.627322] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1273.152909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1278.977712] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1278.989177] Lustre: Skipped 1 previous similar message [ 1279.890565] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1290.324940] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1293.098067] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1299.900381] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1301.491829] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1309.680165] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1312.901473] Lustre: server umount lustre-MDT0000 complete [ 1316.364632] LustreError: 40021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587447 with bad export cookie 14948445773376349423 [ 1316.372618] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1316.380612] LustreError: 40021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1316.688889] Lustre: server umount lustre-MDT0001 complete [ 1331.415756] Lustre: server umount lustre-OST0000 complete [ 1346.180587] Lustre: server umount lustre-OST0001 complete [ 1356.777622] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 01:51:26 (1788587486) [ 1375.416092] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1388.119359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1388.735615] LustreError: 47682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1388.762697] LustreError: 47682:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 32 previous similar messages [ 1394.223815] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1404.307897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1410.230804] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1413.668460] Lustre: 48836:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1420.838863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1428.566258] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1438.501571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1446.030922] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1454.420722] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1460.190712] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1473.319703] Lustre: server umount lustre-MDT0000 complete [ 1477.337782] LustreError: 47663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587608 with bad export cookie 14948445773376350221 [ 1477.353409] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1477.353929] LustreError: 47663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1477.741221] Lustre: server umount lustre-MDT0001 complete [ 1492.543578] Lustre: server umount lustre-OST0000 complete [ 1495.520142] Lustre: 3646:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788587611/real 1788587611] req@ffff88d7f69d8e00 x1875468608840448/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788587627 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1495.549336] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1495.563822] Lustre: Skipped 13 previous similar messages [ 1496.416261] Lustre: server umount lustre-OST0001 complete [ 1507.817059] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 01:53:57 (1788587637) [ 1510.551969] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1512.779173] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 01:54:02 (1788587642) [ 1525.969277] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 1536.222415] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1562.788967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1563.092736] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1563.215041] Lustre: lustre-MDT0000: new disk, initializing [ 1563.341596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1569.017202] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1579.631595] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1586.940271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1587.340143] Lustre: lustre-OST0000: new disk, initializing [ 1587.351506] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1587.359313] Lustre: Skipped 1 previous similar message [ 1587.368429] Lustre: 54907:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1588.822057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1588.830244] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1588.872051] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1595.357050] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1606.918780] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1609.536167] Lustre: DEBUG MARKER: create test files [ 1645.035045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1645.048712] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1645.053560] Lustre: Skipped 22 previous similar messages [ 1650.864482] Lustre: server umount lustre-OST0000 complete [ 1656.074307] Lustre: server umount lustre-MDT0000 complete [ 1669.622634] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1673.190430] Key type lgssc unregistered [ 1673.504207] LNet: 56537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1673.512525] LNetError: 56537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1673.537389] LNet: Removed LNI 192.168.202.159@tcp [ 1674.608267] Key type .llcrypt unregistered [ 1674.611991] Key type ._llcrypt unregistered [ 1702.962058] Key type ._llcrypt registered [ 1702.967529] Key type .llcrypt registered [ 1703.168960] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1704.574977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1704.635045] alg: No test for adler32 (adler32-zlib) [ 1705.831223] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 1706.226504] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1707.968305] Key type lgssc registered [ 1709.419369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1717.592612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1717.619994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1719.124838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1723.913912] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1731.712557] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1739.422657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1739.956589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1744.090181] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1744.252242] Lustre: 59245:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1744.264586] Lustre: 59245:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1744.289690] Lustre: 59245:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1744.295925] Lustre: 59245:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1744.304568] Lustre: 59245:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1744.309915] Lustre: 59245:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1747.555466] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1756.520915] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1763.029287] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1769.912338] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1780.196664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1780.220683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1784.731543] Lustre: server umount lustre-OST0000 complete [ 1790.906415] Lustre: server umount lustre-MDT0000 complete [ 1807.729955] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1811.628276] Key type lgssc unregistered [ 1812.168296] LNet: 60853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1812.173384] LNetError: 60853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1812.200060] LNet: Removed LNI 192.168.202.159@tcp [ 1813.620179] Key type .llcrypt unregistered [ 1813.626520] Key type ._llcrypt unregistered [ 1847.804435] Key type ._llcrypt registered [ 1847.809200] Key type .llcrypt registered [ 1847.957704] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1850.299290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1850.336617] alg: No test for adler32 (adler32-zlib) [ 1851.530431] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 1851.850904] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1853.639280] Key type lgssc registered [ 1855.087130] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1864.420152] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1864.449491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1865.891295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1870.545879] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1878.948875] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1886.500725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1886.805502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1891.970548] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1892.059615] Lustre: 63685:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1892.075206] Lustre: 63685:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1892.087293] Lustre: 63685:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1892.103377] Lustre: 63685:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1892.116054] Lustre: 63685:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1892.123271] Lustre: 63685:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1893.893936] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1900.918694] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1908.030524] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1923.040382] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1923.053260] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1923.060582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1924.546406] Lustre: server umount lustre-OST0000 complete [ 1929.780285] Lustre: server umount lustre-MDT0000 complete [ 1944.495706] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1948.006991] Key type lgssc unregistered [ 1948.316838] LNet: 65256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1948.322395] LNetError: 65256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1948.352728] LNet: Removed LNI 192.168.202.159@tcp [ 1949.376268] Key type .llcrypt unregistered [ 1949.378180] Key type ._llcrypt unregistered [ 1971.715785] Key type ._llcrypt registered [ 1971.717547] Key type .llcrypt registered [ 1971.820953] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 1989.847820] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1991.298693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1991.437565] alg: No test for adler32 (adler32-zlib) [ 1992.641748] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 1992.882394] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1994.560393] Key type lgssc registered [ 1996.027388] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2040.819934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2040.841351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2042.058963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2042.151509] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2042.282690] Lustre: lustre-MDT0000: new disk, initializing [ 2042.381494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2042.417048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2048.828273] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2061.038810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2061.248195] Lustre: 68410:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2061.359800] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2061.366697] Lustre: Skipped 1 previous similar message [ 2061.473476] Lustre: lustre-MDT0001: new disk, initializing [ 2061.625485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2061.672920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2061.700500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2067.029278] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2078.297971] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2085.218815] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2093.426918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2093.787862] Lustre: lustre-OST0000: new disk, initializing [ 2093.792544] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2093.806063] Lustre: 70378:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2093.905984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2099.320460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2099.339504] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2099.521276] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2101.358972] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2112.326719] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2118.195332] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2118.414890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2123.571661] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2123.894268] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2130.402590] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2130.409397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.415218] LustreError: Skipped 1 previous similar message [ 2130.424598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2130.435038] Lustre: Skipped 1 previous similar message [ 2130.451726] Lustre: Skipped 1 previous similar message [ 2132.829492] Lustre: server umount lustre-OST0000 complete [ 2138.592678] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2138.599204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2138.615874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2140.646040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2140.655541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2145.760457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2145.768828] Lustre: Skipped 2 previous similar messages [ 2150.884527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2150.904801] Lustre: Skipped 1 previous similar message [ 2151.391166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2151.538121] Lustre: server umount lustre-MDT0000 complete [ 2155.802846] LustreError: 69217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588287 with bad export cookie 10845644257014798413 [ 2155.815486] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2155.821269] LustreError: 69217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2156.199945] Lustre: server umount lustre-MDT0001 complete [ 2166.267726] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 02:04:56 (1788588296) [ 2178.154435] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2222.875740] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 2235.118056] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2274.174703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2274.392506] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2274.419540] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2274.503940] Lustre: lustre-MDT0000: new disk, initializing [ 2274.573269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2274.589387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2279.715504] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2292.425066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2292.514729] Lustre: 76408:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2292.569405] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2292.573523] Lustre: Skipped 1 previous similar message [ 2292.628446] Lustre: lustre-MDT0001: new disk, initializing [ 2292.724601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2292.784165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2292.798268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2298.205605] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2309.454486] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2315.718768] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2325.205154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2325.616978] Lustre: lustre-OST0000: new disk, initializing [ 2325.620822] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2325.625523] Lustre: 78379:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2325.753198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2330.750966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2330.762364] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2330.985066] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2333.335428] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2346.383308] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2353.675761] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2354.012868] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2359.114589] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2359.312923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2361.828952] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2361.831675] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2361.842922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2361.860545] Lustre: Skipped 1 previous similar message [ 2367.597522] Lustre: server umount lustre-OST0000 complete [ 2372.068356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2372.085347] Lustre: Skipped 1 previous similar message [ 2377.738381] Lustre: server umount lustre-MDT0000 complete [ 2382.073690] LustreError: 76399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588513 with bad export cookie 10845644257014799295 [ 2382.091476] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2382.094285] LustreError: 76399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2382.597956] Lustre: server umount lustre-MDT0001 complete [ 2392.049192] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 02:08:41 (1788588521) [ 2402.561816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2403.778341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2408.953342] LustreError: 80792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2408.972817] LustreError: 80792:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2411.147348] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2414.050059] LustreError: 80793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2415.080728] LustreError: 80792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2420.217257] LustreError: 80792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2422.473454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2429.027784] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2437.603159] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2443.709245] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2451.636143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2452.040919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2452.051629] Lustre: Skipped 1 previous similar message [ 2459.607785] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2467.587416] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2471.354852] Lustre: 83464:0:(genops.c:1774:obd_export_evict_by_uuid()) MGS: evicting 5135ddca-cd7d-4278-a6fe-b35153b71258 at adminstrative request [ 2473.957594] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2473.987296] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2484.199672] Lustre: MGS: Client 5135ddca-cd7d-4278-a6fe-b35153b71258 (at 0@lo) reconnecting [ 2484.211654] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x968377fd339f5f2a to 0x968377fd339f61ca [ 2484.226534] Lustre: MGC192.168.202.159@tcp: Connection restored to 0@lo (at 0@lo) [ 2509.820824] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2510.023941] Lustre: lustre-OST0001: new disk, initializing [ 2510.029939] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2510.046224] Lustre: 83858:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2518.220838] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2519.188307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2519.213424] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2519.406377] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2529.890519] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2534.880875] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2534.884716] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2534.910521] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2534.914294] Lustre: Skipped 1 previous similar message [ 2534.932204] Lustre: Skipped 8 previous similar messages [ 2539.282172] Lustre: server umount lustre-OST0001 complete [ 2540.004036] LustreError: 82616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2540.021273] LustreError: 82616:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2549.071266] Lustre: server umount lustre-OST0000 complete [ 2555.361122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2555.395946] Lustre: Skipped 2 previous similar messages [ 2560.863120] Lustre: server umount lustre-MDT0000 complete [ 2564.578694] LustreError: 80792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2564.600249] LustreError: 80792:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2566.159870] LustreError: 80772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588697 with bad export cookie 10845644257014800842 [ 2566.163733] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2566.168296] LustreError: 80772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2566.750766] Lustre: server umount lustre-MDT0001 complete [ 2577.234573] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 02:11:47 (1788588707) [ 2578.933246] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2581.392640] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 02:11:51 (1788588711) [ 2583.335620] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2585.305760] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 02:11:55 (1788588715) [ 2587.352555] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2589.680257] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 02:11:59 (1788588719) [ 2591.548405] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2593.226749] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 02:12:03 (1788588723) [ 2595.178864] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2601.506501] loop: module loaded [ 2601.538532] loop0: detected capacity change from 0 to 4294967296 [ 2601.582068] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2602.855883] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2606.925531] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2611.466629] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2640.572853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2640.615176] LustreError: 87986:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2640.621758] LustreError: 87986:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2640.629586] LustreError: 87986:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2640.634822] LustreError: 87986:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2663.714697] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 2674.160286] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2713.761539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2714.215873] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2714.243353] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2714.373530] Lustre: lustre-MDT0000: new disk, initializing [ 2714.500175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2714.504362] Lustre: Skipped 1 previous similar message [ 2714.518172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2719.763205] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2731.528861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2731.663301] Lustre: 90854:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2732.067099] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2737.282143] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2747.054880] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2753.137231] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2760.407328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2760.792736] Lustre: 92825:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2766.451302] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2766.575457] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2768.366556] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2779.798354] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2785.377443] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2785.642647] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2791.035934] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2791.267303] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2797.563488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2797.571869] Lustre: Skipped 1 previous similar message [ 2797.578049] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2797.579738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2797.579743] Lustre: Skipped 6 previous similar messages [ 2797.605527] LustreError: Skipped 1 previous similar message [ 2799.816499] Lustre: server umount lustre-OST0000 complete [ 2811.031821] Lustre: server umount lustre-MDT0000 complete [ 2812.906096] LustreError: 90865:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2812.931603] LustreError: 90865:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2815.438585] LustreError: 90845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588947 with bad export cookie 10845644257014801465 [ 2815.447474] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2815.450323] LustreError: 90845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2816.240834] Lustre: server umount lustre-MDT0001 complete [ 2826.461434] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 02:15:56 (1788588956) [ 2834.807483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2840.870809] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2846.180582] LustreError: 95239:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2846.204548] LustreError: 95239:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2849.989154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2850.347745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2850.354276] Lustre: Skipped 3 previous similar messages [ 2855.305210] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2862.885878] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2868.364437] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2874.944518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2882.160646] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2889.711211] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2895.062526] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2895.398889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2899.939872] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2900.151350] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2907.901964] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2908.052288] Lustre: lustre-OST0001: new disk, initializing [ 2908.054718] Lustre: Skipped 2 previous similar messages [ 2908.058728] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2908.061790] Lustre: Skipped 3 previous similar messages [ 2908.066081] Lustre: 98606:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2914.438208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2914.455361] Lustre: Skipped 2 previous similar messages [ 2914.472455] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2914.622214] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2915.408038] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2925.538316] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2930.844328] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2931.235965] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2936.473897] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2936.758287] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2958.350383] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2988.498197] Lustre: Failing over lustre-MDT0000 [ 2988.517040] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2988.532045] LustreError: Skipped 1 previous similar message [ 2988.546018] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2988.563424] Lustre: Skipped 3 previous similar messages [ 2988.591589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2988.595886] Lustre: Skipped 3 previous similar messages [ 2989.077742] Lustre: server umount lustre-MDT0000 complete [ 2990.340347] LustreError: 95233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2990.370197] LustreError: 95233:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3007.954102] Lustre: 66629:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788589123/real 1788589123] req@00000000f11c3692 x1875470581002496/t0(0) o400->MGC192.168.202.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788589139 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3007.977577] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3015.285685] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 3015.289170] LDISKFS-fs (dm-0): recovery complete [ 3015.296661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3018.233821] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x968377fd33a02c4c [ 3018.245946] Lustre: MGC192.168.202.159@tcp: Connection restored to 0@lo (at 0@lo) [ 3018.540917] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3022.050988] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3022.053531] Lustre: 101255:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 3023.110528] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3023.881200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3023.889300] Lustre: 95238:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 3023.921100] Lustre: 95238:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 1 previous similar message [ 3024.029150] LustreError: 101458:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3024.351128] LustreError: 101458:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3024.680543] LustreError: 101458:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3024.691458] LustreError: 101458:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 3025.008603] LustreError: 101458:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3025.025775] LustreError: 101458:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 3025.038932] Lustre: 101458:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3025.046391] Lustre: 101458:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 7 previous similar messages [ 3025.990170] LustreError: 101458:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3026.007819] LustreError: 101458:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 3 previous similar messages [ 3026.327152] LustreError: 101458:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3026.339737] LustreError: 101458:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 3 previous similar messages [ 3027.331441] Lustre: 101458:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3027.347788] Lustre: 101458:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 6 previous similar messages [ 3027.997423] LustreError: 101458:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3028.016303] LustreError: 101458:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 5 previous similar messages [ 3028.343203] LustreError: 101458:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3028.353355] LustreError: 101458:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 5 previous similar messages [ 3031.568296] Lustre: 101458:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3031.604311] Lustre: 101458:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 12 previous similar messages [ 3032.248768] LustreError: 101458:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3032.262489] LustreError: 101458:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 3032.591243] LustreError: 101458:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3032.602250] LustreError: 101458:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 3039.625257] Lustre: 101458:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3039.641089] Lustre: 101458:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 24 previous similar messages [ 3040.281898] LustreError: 101458:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3040.299873] LustreError: 101458:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 24 previous similar messages [ 3040.624084] LustreError: 101458:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3040.635605] LustreError: 101458:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 24 previous similar messages [ 3055.880189] Lustre: 101458:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3055.896116] Lustre: 101458:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 50 previous similar messages [ 3056.554346] LustreError: 101458:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3056.568267] LustreError: 101458:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 50 previous similar messages [ 3056.898276] LustreError: 101458:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3056.919662] LustreError: 101458:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 50 previous similar messages [ 3082.226261] Lustre: 101458:0:(ldlm_lib.c:2419:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 3082.246250] Lustre: 101458:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3082.259688] Lustre: 101458:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client c6041095-f13a-4c8f-ab6f-067b0158b333@192.168.202.59@tcp [ 3082.271328] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3082.277530] Lustre: 101458:0:(ldlm_lib.c:1939:abort_req_replay_queue()) @@@ aborted: req@00000000c909aa83 x1875470561909248/t0(8589934779) o101->c6041095-f13a-4c8f-ab6f-067b0158b333@192.168.202.59@tcp:264/0 lens 592/0 e 0 to 0 dl 1788589224 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 3082.327766] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3082.335029] Lustre: lustre-MDT0000: Denying connection for new client c6041095-f13a-4c8f-ab6f-067b0158b333 (at 192.168.202.59@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 3082.374745] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3082.455583] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3082.465917] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 3082.466749] Lustre: Skipped 2 previous similar messages [ 3082.527406] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:47 to 0x2c0000401:545) [ 3082.533475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:80 to 0x280000401:577) [ 3103.717812] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3108.546149] Lustre: server umount lustre-OST0000 complete [ 3118.666952] Lustre: server umount lustre-OST0001 complete [ 3123.682044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3123.699636] Lustre: Skipped 7 previous similar messages [ 3123.707051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3123.713284] Lustre: Skipped 8 previous similar messages [ 3129.326358] Lustre: server umount lustre-MDT0000 complete [ 3131.360306] LustreError: 95239:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3131.388231] LustreError: 95239:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 45 previous similar messages [ 3133.987412] LustreError: 95220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589265 with bad export cookie 10845644257014852684 [ 3133.989786] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3134.000745] LustreError: 95220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3134.359462] Lustre: server umount lustre-MDT0001 complete [ 3144.005978] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 02:21:14 (1788589274) [ 3153.468915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3153.484957] Lustre: *** cfs_fail_loc=197, val=0*** [ 3169.247711] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3174.367896] LustreError: 103434:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 4: rc = -110 [ 3179.517270] Lustre: *** cfs_fail_loc=197, val=0*** [ 3179.521458] Lustre: Skipped 65 previous similar messages [ 3200.159467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3200.169658] Lustre: Skipped 3 previous similar messages [ 3208.462371] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3218.023199] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3220.499542] Lustre: server umount lustre-OST0000 complete [ 3230.264371] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 02:22:40 (1788589360) [ 3251.837306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3256.918911] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3266.342293] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3270.326482] LustreError: 105087:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3270.337363] LustreError: 105087:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3270.349112] LustreError: 105087:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 3270.951910] Lustre: server umount lustre-MDT0000 complete [ 3308.647490] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 3324.107371] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3367.979871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3368.445906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3368.484313] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3368.643286] Lustre: lustre-MDT0000: new disk, initializing [ 3368.798069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3373.848451] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3387.304826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3387.447814] Lustre: 109282:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3387.731346] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3393.726849] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3406.022973] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3412.638447] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3420.200938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3420.484948] Lustre: 111253:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3426.063911] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3427.880276] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3439.950386] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3445.567745] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3445.749943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3451.451956] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3451.740540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3456.994820] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3456.996548] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3456.999244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3456.999252] Lustre: Skipped 2 previous similar messages [ 3457.020700] Lustre: Skipped 2 previous similar messages [ 3457.057811] LustreError: Skipped 4 previous similar messages [ 3460.426853] Lustre: server umount lustre-OST0000 complete [ 3471.905820] Lustre: server umount lustre-MDT0000 complete [ 3472.352520] LustreError: 109290:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3472.379599] LustreError: 109290:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3476.568502] LustreError: 111252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589608 with bad export cookie 10845644257014863359 [ 3476.588593] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3476.603349] LustreError: 111252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3477.007673] Lustre: server umount lustre-MDT0001 complete [ 3487.346277] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 02:26:57 (1788589617) [ 3498.990133] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3502.185956] Key type lgssc unregistered [ 3502.529081] LNet: 113744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3502.565587] LNetError: 113744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3503.593302] LNet: Removed LNI 192.168.202.159@tcp [ 3504.802848] Key type .llcrypt unregistered [ 3504.807048] Key type ._llcrypt unregistered [ 3527.059264] Key type ._llcrypt registered [ 3527.061989] Key type .llcrypt registered [ 3527.186748] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 3544.169207] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3545.646450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3545.792135] alg: No test for adler32 (adler32-zlib) [ 3546.865472] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 3547.082361] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3548.751234] Key type lgssc registered [ 3549.877783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3597.427304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3597.444540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3598.713693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3598.734080] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3598.860315] Lustre: lustre-MDT0000: new disk, initializing [ 3598.928690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3598.944278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3604.498209] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3616.207333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3616.497633] Lustre: lustre-OST0000: new disk, initializing [ 3616.501357] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3616.505365] Lustre: Skipped 1 previous similar message [ 3616.513909] Lustre: 117926:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3616.604434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3623.617294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3623.638730] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3623.716039] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3625.081787] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3637.764875] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3638.004328] Lustre: lustre-OST0001: new disk, initializing [ 3638.007672] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3638.014737] Lustre: 118790:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3638.098634] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3644.135216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3644.151533] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3644.182516] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3646.002745] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3659.749035] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3659.776149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3661.647317] Lustre: server umount lustre-MDT0000 complete [ 3669.241303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3674.591897] LustreError: 117113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589806 with bad export cookie 1705621359281459920 [ 3674.601533] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3674.610110] LustreError: 117113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3674.709647] Lustre: server umount lustre-OST0000 complete [ 3679.006204] Lustre: server umount lustre-OST0001 complete [ 3689.211280] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 02:30:19 (1788589819) [ 3717.353523] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 02:30:47 (1788589847) [ 3728.647586] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3755.244730] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 02:31:25 (1788589885) [ 3802.628252] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 3814.212435] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3860.382513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3860.622654] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3860.666817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3860.752916] Lustre: lustre-MDT0000: new disk, initializing [ 3860.858528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3860.895487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3865.955426] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3876.796040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3876.894494] Lustre: 126548:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3876.931940] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3876.934615] Lustre: Skipped 1 previous similar message [ 3876.977621] Lustre: lustre-MDT0001: new disk, initializing [ 3877.113742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3877.180179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3877.193144] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3883.803836] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3896.831962] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3905.384969] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3912.879083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3913.190963] Lustre: lustre-OST0000: new disk, initializing [ 3913.196455] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3913.202323] Lustre: 128516:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3913.295746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3918.948772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3918.959154] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3919.155664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3919.336299] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3929.894514] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3934.909483] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3935.109418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3940.173378] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3940.460244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3944.939609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3944.944326] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3944.953662] LustreError: Skipped 1 previous similar message [ 3944.956437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3944.963219] Lustre: Skipped 2 previous similar messages [ 3944.974098] Lustre: Skipped 2 previous similar messages [ 3948.652806] Lustre: server umount lustre-OST0000 complete [ 3954.145813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3954.153444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3954.177191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3958.605829] Lustre: server umount lustre-MDT0000 complete [ 3961.934794] LustreError: 127343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590093 with bad export cookie 1705621359281461138 [ 3961.936274] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3961.949160] LustreError: 127343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3962.416882] Lustre: server umount lustre-MDT0001 complete [ 3971.236458] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 02:35:01 (1788590101) [ 3978.824778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3979.194386] LustreError: 130928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3979.210121] LustreError: 130928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3979.264446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3983.388161] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3984.360761] LustreError: 130929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3988.467846] LustreError: 130928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3991.856359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3997.377556] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4005.928300] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4012.578364] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4020.235516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4020.614352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4020.634371] Lustre: Skipped 1 previous similar message [ 4027.213318] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4035.449777] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4043.631566] Lustre: *** cfs_fail_loc=159, val=0*** [ 4059.930092] Lustre: lustre-MDT0000: Client f1310a6b-4c93-473e-9f24-931d818f925b (at 192.168.202.59@tcp) reconnecting [ 4061.621816] Lustre: *** cfs_fail_loc=159, val=0*** [ 4061.626881] Lustre: Skipped 5 previous similar messages [ 4077.331689] Lustre: lustre-MDT0000: Client f1310a6b-4c93-473e-9f24-931d818f925b (at 192.168.202.59@tcp) reconnecting [ 4083.171419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4083.187181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4083.189734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4083.204804] Lustre: Skipped 2 previous similar messages [ 4083.220936] Lustre: Skipped 3 previous similar messages [ 4088.298976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4088.320276] Lustre: Skipped 1 previous similar message [ 4088.688655] Lustre: server umount lustre-OST0000 complete [ 4093.427798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4093.447466] Lustre: Skipped 1 previous similar message [ 4098.531552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4098.541119] Lustre: Skipped 3 previous similar messages [ 4098.777468] Lustre: server umount lustre-MDT0000 complete [ 4102.494568] LustreError: 130908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590234 with bad export cookie 1705621359281462020 [ 4102.495946] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4102.504979] LustreError: 130908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4102.934526] Lustre: server umount lustre-MDT0001 complete [ 4113.511760] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 4116.365214] Key type lgssc unregistered [ 4116.732564] LNet: 134701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4116.740110] LNetError: 134701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4116.752833] LNet: Removed LNI 192.168.202.159@tcp [ 4117.587704] Key type .llcrypt unregistered [ 4117.589720] Key type ._llcrypt unregistered [ 4144.211640] Key type ._llcrypt registered [ 4144.217267] Key type .llcrypt registered [ 4144.361852] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4146.058472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4146.111382] alg: No test for adler32 (adler32-zlib) [ 4147.258592] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 4147.488254] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 4149.167236] Key type lgssc registered [ 4150.590076] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4159.561477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4159.581676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4161.203466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4165.938800] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4170.753918] LustreError: 136114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4170.771970] LustreError: 136114:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4175.844300] LustreError: 136115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4175.861373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4176.256114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4182.173901] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4191.412765] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4198.116609] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4206.805558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4207.445373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4215.427648] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4225.168341] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4231.095833] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4240.783384] Lustre: *** cfs_fail_loc=159, val=0*** [ 4257.106341] Lustre: lustre-MDT0000: Client 6e9b7f02-4c7a-48c2-a407-0072a7ef229c (at 192.168.202.59@tcp) reconnecting [ 4259.709570] Lustre: *** cfs_fail_loc=159, val=0*** [ 4259.721116] Lustre: Skipped 1 previous similar message [ 4276.005081] Lustre: lustre-MDT0001: Client 6e9b7f02-4c7a-48c2-a407-0072a7ef229c (at 192.168.202.59@tcp) reconnecting [ 4278.087909] Lustre: *** cfs_fail_loc=159, val=0*** [ 4278.103721] Lustre: Skipped 2 previous similar messages [ 4293.427982] Lustre: lustre-MDT0001: Client 6e9b7f02-4c7a-48c2-a407-0072a7ef229c (at 192.168.202.59@tcp) reconnecting [ 4298.914597] Lustre: 139159:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4302.018095] Lustre: *** cfs_fail_loc=159, val=0*** [ 4302.023285] Lustre: Skipped 7 previous similar messages [ 4317.489023] Lustre: lustre-MDT0000: Client 8d12aafb-c5a6-4ca1-84dc-aeb15a19b30b (at 192.168.202.59@tcp) reconnecting [ 4319.067687] Lustre: *** cfs_fail_loc=159, val=0*** [ 4319.069563] Lustre: Skipped 11 previous similar messages [ 4334.390137] Lustre: lustre-MDT0000: Client 8d12aafb-c5a6-4ca1-84dc-aeb15a19b30b (at 192.168.202.59@tcp) reconnecting [ 4337.347601] Lustre: 139451:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4347.442434] Lustre: server umount lustre-OST0000 complete [ 4354.037020] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4354.048882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4354.057208] Lustre: Skipped 1 previous similar message [ 4354.074275] Lustre: Skipped 1 previous similar message [ 4359.137538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4362.377961] Lustre: server umount lustre-MDT0000 complete [ 4364.258580] LustreError: 136842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4364.284868] LustreError: 136842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4365.290565] Lustre: 135425:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788590480/real 1788590480] req@00000000728eb1ce x1875472840472448/t0(0) o6->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788590496 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4365.292617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4365.325392] Lustre: 135425:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4366.364053] LustreError: 139398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590497 with bad export cookie 3487121866007663942 [ 4366.369649] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4366.377632] LustreError: 139398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4366.907881] Lustre: server umount lustre-MDT0001 complete [ 4378.677872] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 4381.997693] Key type lgssc unregistered [ 4382.408848] LNet: 140507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4382.417398] LNetError: 140507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4382.438235] LNet: Removed LNI 192.168.202.159@tcp [ 4383.557374] Key type .llcrypt unregistered [ 4383.559494] Key type ._llcrypt unregistered [ 4411.029306] Key type ._llcrypt registered [ 4411.031432] Key type .llcrypt registered [ 4411.160417] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4412.793169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4412.861413] alg: No test for adler32 (adler32-zlib) [ 4414.050149] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 4414.398611] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 4416.087189] Key type lgssc registered [ 4417.783202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4427.156416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4427.175443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4428.813619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4434.338868] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4439.027144] LustreError: 141919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4439.040115] LustreError: 141919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4444.131401] LustreError: 141920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4444.742739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4445.267655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4450.150383] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4458.116990] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4465.193458] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4472.388901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4472.682655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4473.739217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4479.091849] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4487.376720] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4490.115931] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4494.532769] Lustre: 144505:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4497.680862] Lustre: 144555:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4499.169689] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4500.307325] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4505.533057] Lustre: 144702:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4506.704079] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4507.761681] Lustre: 144800:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4514.787304] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4514.791963] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4514.800867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4515.816088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4515.819700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4515.825588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4519.126783] Lustre: server umount lustre-OST0000 complete [ 4526.055138] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4526.066977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4526.069700] Lustre: Skipped 1 previous similar message [ 4531.170328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4531.175180] Lustre: Skipped 1 previous similar message [ 4536.289807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4536.293635] Lustre: Skipped 1 previous similar message [ 4538.335123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4538.588291] Lustre: server umount lustre-MDT0000 complete [ 4541.409795] LustreError: 141919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4541.430936] LustreError: 141919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4543.123266] LustreError: 141901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590674 with bad export cookie 403131771229022447 [ 4543.135639] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4543.145089] LustreError: 141901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4543.726496] Lustre: server umount lustre-MDT0001 complete [ 4556.121984] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 4559.589102] Key type lgssc unregistered [ 4559.953209] LNet: 145862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4559.969469] LNetError: 145862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4559.999968] LNet: Removed LNI 192.168.202.159@tcp [ 4560.911448] Key type .llcrypt unregistered [ 4560.915405] Key type ._llcrypt unregistered [ 4586.313400] Key type ._llcrypt registered [ 4586.318452] Key type .llcrypt registered [ 4586.505517] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4587.948331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4588.373236] alg: No test for adler32 (adler32-zlib) [ 4589.606135] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 4590.015864] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 4591.735213] Key type lgssc registered [ 4593.288247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4602.777739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4602.817232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4604.458672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4610.308690] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4613.615228] LustreError: 147276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4613.628751] LustreError: 147276:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4618.723428] LustreError: 147277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4620.238765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4620.689680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4626.688461] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4635.989654] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4643.017695] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4651.746865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4652.292345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4659.961592] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4665.622865] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4668.316876] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4673.796055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4680.580735] Lustre: *** cfs_fail_loc=159, val=0*** [ 4696.365296] Lustre: lustre-MDT0000: Client 741691a3-cb8e-4343-8dfa-32080674a185 (at 192.168.202.59@tcp) reconnecting [ 4701.666793] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4701.673163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4701.686945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4702.184018] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4702.187096] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4702.196451] LustreError: Skipped 1 previous similar message [ 4702.238347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4703.894649] Lustre: server umount lustre-OST0000 complete [ 4707.808897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4707.814270] LustreError: Skipped 3 previous similar messages [ 4707.826384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4707.848027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4709.870183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4709.881912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4714.073847] Lustre: server umount lustre-MDT0000 complete [ 4714.977249] LustreError: 148008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4715.005556] LustreError: 148008:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4718.346312] LustreError: 148018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590849 with bad export cookie 3548079272637316439 [ 4718.347842] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4718.360411] LustreError: 148018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4718.722522] Lustre: server umount lustre-MDT0001 complete [ 4730.330965] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 4732.965904] Key type lgssc unregistered [ 4733.243592] LNet: 150962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4733.251222] LNetError: 150962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4734.312640] LNet: Removed LNI 192.168.202.159@tcp [ 4735.456087] Key type .llcrypt unregistered [ 4735.457465] Key type ._llcrypt unregistered [ 4758.963819] Key type ._llcrypt registered [ 4758.970921] Key type .llcrypt registered [ 4759.099717] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4760.225749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4760.329688] alg: No test for adler32 (adler32-zlib) [ 4761.396584] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 4761.663838] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 4763.319224] Key type lgssc registered [ 4764.355254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4772.615279] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4772.629439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4774.185518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4778.471225] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4783.594673] LustreError: 152390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4783.629097] LustreError: 152390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4787.127639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4787.739892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4792.711368] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4800.580734] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4806.394925] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4814.162991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4814.496478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4818.620291] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4821.632772] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4829.970900] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4832.966727] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4855.435150] Lustre: lustre-MDT0001: evicting 7c497f5b-e196-436f-809e-c50c083f259b (at 192.168.202.59@tcp) by administrative request [ 4855.443642] Lustre: Skipped 1 previous similar message [ 4885.472449] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4885.495636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4885.518474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4889.569751] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4889.586593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4889.600863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4894.705091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4894.714312] Lustre: Skipped 2 previous similar messages [ 4899.295224] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4899.422731] Lustre: server umount lustre-OST0000 complete [ 4904.932933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4904.936845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4904.959489] Lustre: Skipped 1 previous similar message [ 4904.978210] Lustre: Skipped 1 previous similar message [ 4910.051079] LustreError: 152389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4910.054569] Lustre: server umount lustre-MDT0000 complete [ 4910.059707] LustreError: 152389:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4913.654519] LustreError: 152371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788591045 with bad export cookie 7124091663570635750 [ 4913.664861] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4913.668623] LustreError: 152371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4914.091523] Lustre: server umount lustre-MDT0001 complete [ 4926.712352] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 4929.607638] Key type lgssc unregistered [ 4930.017773] LNet: 156461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4930.023902] LNetError: 156461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4930.048583] LNet: Removed LNI 192.168.202.159@tcp [ 4930.842475] Key type .llcrypt unregistered [ 4930.845267] Key type ._llcrypt unregistered [ 5014.156951] Key type ._llcrypt registered [ 5014.159034] Key type .llcrypt registered [ 5014.259481] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5015.487750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5015.625169] alg: No test for adler32 (adler32-zlib) [ 5016.713850] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 5016.976263] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 5018.711662] Key type lgssc registered [ 5019.758757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5028.121118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5028.151905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5029.676207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5033.924478] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5040.102468] LustreError: 158950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5040.130805] LustreError: 158950:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5043.313939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5043.920420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5048.594506] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5057.609756] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5063.944949] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5071.564175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5072.257359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5075.365544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 5080.187602] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5089.669743] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 5090.239882] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5101.034721] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5101.059117] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5101.083589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5105.140842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5105.149433] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5105.150397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5106.847213] Lustre: server umount lustre-OST0000 complete [ 5115.371689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5115.378662] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5115.403523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5115.415678] Lustre: Skipped 2 previous similar messages [ 5117.893916] Lustre: server umount lustre-MDT0000 complete [ 5120.480237] LustreError: 158950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5120.507032] LustreError: 158950:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5122.782752] LustreError: 158930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788591254 with bad export cookie 9485615893615183373 [ 5122.790934] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5122.813923] LustreError: 158930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5123.279519] Lustre: server umount lustre-MDT0001 complete [ 5137.133677] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5140.792559] Key type lgssc unregistered [ 5141.057057] LNet: 162492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5141.061277] LNetError: 162492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5141.085334] LNet: Removed LNI 192.168.202.159@tcp [ 5141.802387] Key type .llcrypt unregistered [ 5141.805581] Key type ._llcrypt unregistered [ 5169.344776] Key type ._llcrypt registered [ 5169.349637] Key type .llcrypt registered [ 5169.510856] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5170.752050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5170.791242] alg: No test for adler32 (adler32-zlib) [ 5172.114215] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 5172.507535] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 5174.215166] Key type lgssc registered [ 5175.777871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5192.493474] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 02:55:22 (1788591322) [ 5209.260152] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 5223.260788] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5266.059743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5266.125866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5267.488031] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5267.524401] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5267.627421] Lustre: lustre-MDT0000: new disk, initializing [ 5267.730947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5267.749928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5273.609578] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5286.386275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5286.552211] Lustre: 166753:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5286.624367] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5286.630068] Lustre: Skipped 1 previous similar message [ 5286.741861] Lustre: lustre-MDT0001: new disk, initializing [ 5286.920401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5287.008496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5287.026429] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5292.434428] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5303.720956] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5311.194274] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5320.390932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5320.752476] Lustre: lustre-OST0000: new disk, initializing [ 5320.758573] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5320.767275] Lustre: 168725:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5320.864222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5325.916032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5325.928291] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5326.101314] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5328.413634] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5340.274401] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5351.906584] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5351.910445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5351.924997] Lustre: Skipped 1 previous similar message [ 5351.961715] Lustre: Skipped 1 previous similar message [ 5357.028360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5357.037029] Lustre: Skipped 1 previous similar message [ 5357.929389] Lustre: server umount lustre-OST0000 complete [ 5362.145437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5362.148442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5362.179035] Lustre: Skipped 1 previous similar message [ 5362.203984] Lustre: Skipped 1 previous similar message [ 5367.268655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5367.272695] Lustre: Skipped 1 previous similar message [ 5368.112923] Lustre: server umount lustre-MDT0000 complete [ 5371.497485] LustreError: 166746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788591503 with bad export cookie 11803520837963283848 [ 5371.498789] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5371.507743] LustreError: 166746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5371.853331] Lustre: server umount lustre-MDT0001 complete [ 5382.442831] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5384.829777] Key type lgssc unregistered [ 5385.112930] LNet: 170632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5385.117881] LNetError: 170632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5386.151827] LNet: Removed LNI 192.168.202.159@tcp [ 5387.087514] Key type .llcrypt unregistered [ 5387.089568] Key type ._llcrypt unregistered [ 5410.347295] Key type ._llcrypt registered [ 5410.349191] Key type .llcrypt registered [ 5410.436930] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5412.055183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5412.122711] alg: No test for adler32 (adler32-zlib) [ 5413.204857] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 5413.579528] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 5415.367252] Key type lgssc registered [ 5417.058761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5424.567128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5424.582830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5425.927357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5430.912971] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5436.393306] LustreError: 172041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5436.407071] LustreError: 172041:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5440.709575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5441.113408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5446.214665] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5455.509927] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5461.734241] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5469.619479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5469.929956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5470.980737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5477.060831] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5485.529083] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5492.847092] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5492.873447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5492.898669] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5492.906762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.919568] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5493.989681] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5494.007467] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5494.024487] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5494.033763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.049197] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5495.220586] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5495.240994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5495.252830] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5495.261601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.280072] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5496.319495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.227643] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5497.235548] Lustre: Skipped 1 previous similar message [ 5497.247805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5497.255508] Lustre: Skipped 1 previous similar message [ 5497.266535] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5497.272058] Lustre: Skipped 1 previous similar message [ 5497.279033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.291670] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5497.309507] Lustre: Skipped 1 previous similar message [ 5498.339481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.267646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.520185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.732962] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5501.742487] Lustre: Skipped 3 previous similar messages [ 5501.784986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5501.796042] Lustre: Skipped 3 previous similar messages [ 5501.800618] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5501.807422] Lustre: Skipped 3 previous similar messages [ 5501.815476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.829994] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5501.840203] Lustre: Skipped 3 previous similar messages [ 5502.964510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.907127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.204792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.550248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.590850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.761212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.715930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.689919] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5510.697338] Lustre: Skipped 7 previous similar messages [ 5510.717794] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5510.725361] Lustre: Skipped 7 previous similar messages [ 5510.728408] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5510.730905] Lustre: Skipped 7 previous similar messages [ 5510.734780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.739627] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5510.742299] Lustre: Skipped 7 previous similar messages [ 5511.880337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.725467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.816982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.886250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.130689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.187787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.115396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.302238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.283889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.413815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.569378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.469732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.631599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.567588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.631605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.499828] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5527.505353] Lustre: Skipped 15 previous similar messages [ 5527.521802] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5527.533037] Lustre: Skipped 15 previous similar messages [ 5527.536978] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5527.539882] Lustre: Skipped 15 previous similar messages [ 5527.545853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.558937] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5527.561787] Lustre: Skipped 15 previous similar messages [ 5528.653040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.900093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.158610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.136214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.138084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.155535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.348507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.296606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.523712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.900269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.207742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.252536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.249633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.315741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.355720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.601863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.838260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.943719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.120879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.158886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.392675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.190403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.305192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.345126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.470553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.517526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.669692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.902614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.132492] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5560.139233] Lustre: Skipped 28 previous similar messages [ 5560.158583] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5560.172139] Lustre: Skipped 28 previous similar messages [ 5560.177561] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5560.185786] Lustre: Skipped 28 previous similar messages [ 5560.190214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.198978] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5560.209321] Lustre: Skipped 28 previous similar messages [ 5561.175488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.258546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.251822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.406278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.765110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.918294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.885661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.145288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.871334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.553531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.381918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.470938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.667776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.606274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.747531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.408423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.438833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.397566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.988172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.520685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.076820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.478677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.969810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.825403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.569500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.457781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.571272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.902628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.353631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.677627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.911873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.839369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.772982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.872684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.156623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.361892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.230642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.012946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.231231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.256464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.211777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.963095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.948981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.328836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.447685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.402221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.256282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.196715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.432142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.316934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.316294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.370893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.385817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.305063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.290666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.134770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.084750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.123323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.126786] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5625.141637] Lustre: Skipped 58 previous similar messages [ 5625.184886] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5625.199265] Lustre: Skipped 58 previous similar messages [ 5625.208379] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5625.211027] Lustre: Skipped 58 previous similar messages [ 5625.217876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.227466] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5625.243137] Lustre: Skipped 58 previous similar messages [ 5626.295806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.334418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.177486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.148850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.518874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.569617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.592877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.773984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.707758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.824579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.834892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.827046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.802644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.220823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.391673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.373179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.451610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.478349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.677253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.857053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.239885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.496913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.459472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.566524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.941254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.003945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.792429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.108792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.639989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.664833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.891261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.055671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.841728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.586642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.329124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.305296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.224548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.335314] LustreError: 172061:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5666.354852] LustreError: 172061:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5666.363128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.338850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.509147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.291560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.170276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.152444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.185548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.365474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.857193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.031809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.079831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.778804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.790663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.979758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.953688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.943699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.794853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.693976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.564852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.444586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.263753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.364693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.345657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.442207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.274500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.326995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.477547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.530850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.797854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.759152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.669091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.745181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.735493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.780219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.766572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.440068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.085305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.057436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.218705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.463797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.500653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.550963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.841419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.038796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.739510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.334081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.659171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.736677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.890127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.029128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.950325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.794745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.695540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.780078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.806258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.814658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.741720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.045634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.106984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.799917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.758653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.490820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.040793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.144995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.277401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.242663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.580453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.898609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.083721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.275087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.404954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.641523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.555426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.708711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.954638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.816565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.989731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.008826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.087843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.281670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.409568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.480059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.648981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.594069] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5753.597678] Lustre: Skipped 120 previous similar messages [ 5753.613426] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5753.620859] Lustre: Skipped 120 previous similar messages [ 5753.624401] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5753.627655] Lustre: Skipped 120 previous similar messages [ 5753.633586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.643570] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5753.653248] Lustre: Skipped 120 previous similar messages [ 5754.714770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.896259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.184337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.518699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.541815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.822517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.861775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.034311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.306469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.151312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.336399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.377899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.246556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.164246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.331436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.265898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.132289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.170991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.407993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.324245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.281844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.179497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.241708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.146216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.123610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.967982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.181091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.394261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.322179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.679725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.787622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.871921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.882946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.051872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.002221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.246048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.119062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.451355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.551548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.645941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.609979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.610220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.845054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.073205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.167870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.452212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.482765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.747832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.726042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.900046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.907783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.212338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.969974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.449760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.614650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.005706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.939588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.264771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.421149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.559099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.570866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.688550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.949425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.253518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.830103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.034046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.124145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.270781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.487745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.741925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.857346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.027650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.994480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.045058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.137835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.163927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.313233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.477296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.519402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.729209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.959401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.026363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.088800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.244965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.236509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.152067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.109970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.093432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.193742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.356275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.347303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.336981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.283201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.471628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.349299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.223666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.153780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.314465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.557525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.408277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.377908] LustreError: 172061:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5864.396355] LustreError: 172061:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5864.398195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.412739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.505653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.389193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.552216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.782552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.668878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.639950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.798507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.005979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.241793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.371629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.312771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.266106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.094880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.230605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.289349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.439058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.495404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.604249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.411726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.550984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.398296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.326933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.094644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.247493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.988839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.974970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.164569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.348757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.761327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.565628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.515702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.091369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.538127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.163910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.606976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.095094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.121328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.238733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.048981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.050654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.094854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.691578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.850432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.842603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.919575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.133468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.136972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.350946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.706886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.858779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.289955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.394419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.469928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.433567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.475566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.397823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.703873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.660981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.858869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.810891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.867354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.037612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.866721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.779522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.800359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.829317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.749154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.790821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.857556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.799151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.939819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.209846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.131682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.062947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.985067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.118584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.115958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.017552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.851791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.802576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.928156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.993708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.899861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.939968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.167125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.202505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.565687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.954458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.976301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.136899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.228736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.617692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.763177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.318531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.982983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.870498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.101304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.389113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.327532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.754780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.738844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.541481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.746430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.662658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.432813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.261461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.364261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.527137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.551365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.593696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.602981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.616696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.784166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.866845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.723767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.651974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.794112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.742392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.718822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.623687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.535918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.397390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.287604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.464555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.378833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.419509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.401769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.451458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.457680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.365042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.340964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.245497] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6010.249801] Lustre: Skipped 233 previous similar messages [ 6010.266163] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6010.279499] Lustre: Skipped 233 previous similar messages [ 6010.289043] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6010.293704] Lustre: Skipped 233 previous similar messages [ 6010.296830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.306065] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6010.308652] Lustre: Skipped 233 previous similar messages [ 6011.276760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.179432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.047354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.016601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.050325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.251688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.158031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.049453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.020629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.792407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.667102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.680473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.672543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.632719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.718644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.814932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.909552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.017378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.996452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.923734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.906868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.942338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.807681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.895140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.891405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.927419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.878706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.907048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.838706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.851059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.893446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.984411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.897994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.851677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.751754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.691858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.670800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.675586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.851055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.915506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.794907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.919513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.987524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.032516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.097479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.874914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.719579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.612589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.577741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.453218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.397491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.613338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.549986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.409505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.285604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.328589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.434770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.468709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.524605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.468494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.514185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.277947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.155772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.146172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.092878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.930955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.726338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.647695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.502181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.601842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.556344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.397957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.258863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.976697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.983926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.791504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.634463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.631551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.573873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.623799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.621695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.851970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.771649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.761792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.763279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.800779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.642983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.435495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.301966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.133769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.763404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.736165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.618850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.659681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.751379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.807303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.801272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.922485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.797206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.768881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.726067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.672513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.560994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.426917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.527473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.617689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.701254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.776904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.870635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.659786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.575145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.559881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.426051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.357754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.678661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.596286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.666171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.666948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.493989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.433925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.348332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.256325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6129.231332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.268063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.175976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.160950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.333971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.296369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.198956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.136929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.035049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.063641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.064957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6140.028676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6140.972828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.081183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.074861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.960810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.970943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.983341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.787606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.721103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.596396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.391401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6150.153075] LustreError: 172061:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6150.159368] LustreError: 172061:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6150.170887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6150.922058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.657191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.488816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6153.314682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6154.185076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.047692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.787310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.753507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6157.595394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6158.649163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6159.878120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6160.544126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.620459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.964363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6163.970700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.177645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.204301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6167.208297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6168.317015] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6169.427624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6170.500609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6171.590090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6172.582711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6173.795853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.761517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6175.816825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6176.885790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.926239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6178.973852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6180.085434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6181.021246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6182.083445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.015323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.269795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6185.234947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6186.178729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.069163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.168779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6189.190280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.091665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.128446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6192.154492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6193.366849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.289486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.248688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.173252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.916421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.815176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.713388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6199.986123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6200.817248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6201.936647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.113790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6204.038652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6205.026311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6205.879182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.759496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6207.626078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6208.502888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6209.522257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6210.425637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6211.382776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6212.325177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6213.351208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6214.277521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6215.137681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6216.188436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6217.195259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6218.117589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6219.054180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6219.953966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6220.614849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6221.544742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6222.377717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6223.262355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6224.173034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6225.163936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6226.034614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6226.937747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6227.881555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6228.733675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6229.695245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6230.643538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6231.347464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6232.297669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6233.167780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6234.295974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6235.234824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6236.283779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6237.327255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6238.530597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6239.572637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6240.546826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6241.457482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6242.392512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6243.334701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6244.245884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6245.223704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6246.108896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6247.206860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6248.028699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6249.141085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6250.294760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6251.397987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6252.419290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6253.456588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6254.465276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6255.620681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6256.724675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6257.933057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6258.843730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6259.742101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6260.764576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6261.651843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6262.643079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6263.554228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6264.557316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6265.169408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6266.182143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6267.144100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6267.842755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6268.679745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6269.487722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6270.587198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6271.454294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6272.395913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6273.243805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6274.349686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6275.610080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6276.700111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6277.990106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6279.329311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6280.110547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6281.532843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6282.868725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6283.972673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6284.878983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6285.915300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6287.025501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6288.248048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6289.256578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6290.166145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6291.293263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6292.367668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6293.393053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6294.415222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6295.753808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6296.637292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6297.661606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6298.554190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6299.463468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6300.308846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6301.264831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6302.170862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6303.201630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6304.020833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6304.958216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6305.971879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6307.068651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6308.224062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6309.262793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6310.131347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6311.084973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6311.866921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6312.752666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6313.692903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6314.586100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6315.569534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6316.557398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6317.604757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6318.509660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6319.499810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6320.512295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6321.440926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6322.331145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6323.230575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6324.078147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6324.901972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6325.740206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6326.807640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6327.654752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6328.648440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6329.598444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6330.563121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6331.412209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6332.222341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6333.116227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6334.134110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6335.029799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6336.033661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6336.837846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6337.763675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6338.516349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6339.350262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6340.189816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6341.037156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6341.977053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6343.071959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6344.278604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6345.558919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6346.765836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6347.781622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6348.828058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6349.795863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6350.878655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6351.916850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6352.789827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6353.913200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6354.797510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6355.659763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6356.517775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6357.235675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6358.165888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6358.883984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6359.783300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6360.725852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6361.656210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6362.558873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6363.434339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6364.124119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6365.119492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6365.942173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6366.865240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6367.871399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6368.849906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6369.837945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6370.691452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6371.530908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6372.561198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6373.495114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6374.344762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6375.329917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6376.356065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6377.277225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6378.266824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6379.160992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6379.901073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6380.826421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6381.831298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6382.771215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6383.640061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6384.666881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6385.603588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6386.640439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6387.572475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6388.541373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6389.504039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6390.355372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6391.445823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6392.512638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6393.361115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6394.149188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6394.880930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6395.658775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6396.426960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6397.228648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6398.030775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6399.147916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6400.054103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6401.096954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6401.958709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6402.882155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6403.615427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6404.393519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6405.457210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6406.211943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6407.043651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6408.041335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6408.851978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6409.765815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6410.683163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6411.686053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6412.576455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6413.454425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6414.549549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6415.664622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6416.771640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6417.855394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6419.077931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6420.011512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6420.944299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6421.959153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6422.721230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6423.514870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6424.451511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6425.264705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6426.021100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6426.776960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6427.733980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6428.667959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6429.620097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6430.600815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6431.548334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6432.372496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6433.210109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6433.891920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6434.806188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6435.714810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6436.547862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6437.467209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6438.519413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6439.383887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6440.502918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6441.597684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6442.742840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6443.688309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6444.714910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6445.667828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6446.763688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6447.849136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6448.880426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6449.975840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6450.907796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6451.862754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6452.718624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6453.637900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6454.621247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6455.396342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6456.179905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6457.040879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6457.778571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6458.752806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6459.559692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6460.663249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6461.494430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6462.433695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6463.315831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6464.255274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6465.258875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6466.244708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6467.322939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6468.345898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6469.344364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6470.315939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6471.401086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6472.311327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6473.255924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6474.114952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6475.077963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6476.031784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6476.894895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6477.766682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6478.483696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6479.206862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6480.061899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6480.953494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6482.026401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6483.025847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6483.904518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6484.737771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6485.820849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6486.805746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6487.608166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6488.774713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6489.718319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6490.694506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6491.491408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6492.640846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6493.533303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6494.641879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6495.509403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6496.517768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6497.542688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6498.570489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6499.262575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6500.370149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6501.284520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6502.365660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6503.344544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6504.492216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6505.579847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6506.569645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6507.635111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6508.749362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6510.065494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6510.913179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6511.691322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6512.548710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6513.673512] LustreError: 172061:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6513.688264] LustreError: 172061:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6513.704465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6914.258390] LustreError: 173865:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6914.281203] LustreError: 172061:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 7589.856263] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7589.861496] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7589.865216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7589.879369] Lustre: Skipped 525 previous similar messages [ 7594.999523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7595.005507] Lustre: Skipped 1 previous similar message [ 7595.990734] Lustre: server umount lustre-OST0000 complete [ 7600.106527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7605.217410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7605.229257] Lustre: Skipped 2 previous similar messages [ 7606.804531] Lustre: server umount lustre-MDT0000 complete [ 7610.340995] LustreError: 172770:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7610.358856] LustreError: 172770:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7611.072607] LustreError: 172777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593742 with bad export cookie 16207368052867527769 [ 7611.085729] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7611.098318] LustreError: 172777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7611.865673] Lustre: server umount lustre-MDT0001 complete [ 7623.917532] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 7626.634474] Key type lgssc unregistered [ 7626.878557] LNet: 227324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7626.889341] LNetError: 227324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7626.902054] LNet: Removed LNI 192.168.202.159@tcp [ 7627.531726] Key type .llcrypt unregistered [ 7627.534115] Key type ._llcrypt unregistered [ 7653.568608] Key type ._llcrypt registered [ 7653.572574] Key type .llcrypt registered [ 7653.704691] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 7654.804718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7654.872690] alg: No test for adler32 (adler32-zlib) [ 7656.430354] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 7656.865864] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 7658.631190] Key type lgssc registered [ 7659.973251] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7669.220591] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7669.261820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7670.797385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7676.466554] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7680.998986] LustreError: 228736:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7681.016216] LustreError: 228736:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7685.633541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7686.044477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7690.690073] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7699.629955] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7706.813701] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7716.409598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7716.920165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7725.732024] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7734.342878] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7740.561039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7744.412940] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7746.375577] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7746.389071] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7746.408972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7746.428699] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7746.429422] LustreError: 228756:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7751.144380] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7751.157720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7751.169584] Lustre: Skipped 1 previous similar message [ 7751.184765] Lustre: Skipped 1 previous similar message [ 7754.866907] Lustre: server umount lustre-OST0000 complete [ 7766.496641] LustreError: 228756:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7789.470152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7789.948788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7790.993477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7791.016296] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7791.018831] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7791.039690] Lustre: Skipped 1 previous similar message [ 7796.312725] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7806.034283] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7807.815271] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7814.955301] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff94e109a4d800.ost_server_uuid 50 [ 7816.811682] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff94e109a4d800.ost_server_uuid in IDLE state after 0 sec [ 7821.792620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7821.798505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7821.810549] Lustre: Skipped 1 previous similar message [ 7821.821422] Lustre: Skipped 2 previous similar messages [ 7824.110316] Lustre: server umount lustre-OST0000 complete [ 7829.472722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7829.482453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7829.506087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7832.033716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7832.052212] Lustre: Skipped 1 previous similar message [ 7837.152707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7837.168448] Lustre: Skipped 1 previous similar message [ 7843.295701] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7843.436729] Lustre: server umount lustre-MDT0000 complete [ 7847.395858] LustreError: 228733:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7847.424510] LustreError: 228733:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7848.325490] LustreError: 228715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593979 with bad export cookie 2031456874684019566 [ 7848.337042] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7848.339712] LustreError: 228715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7848.815366] Lustre: server umount lustre-MDT0001 complete [ 7864.782753] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 7867.883159] Key type lgssc unregistered [ 7868.281358] LNet: 233948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7868.297198] LNetError: 233948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7868.340532] LNet: Removed LNI 192.168.202.159@tcp [ 7869.485297] Key type .llcrypt unregistered [ 7869.487909] Key type ._llcrypt unregistered [ 7910.797828] Key type ._llcrypt registered [ 7910.800775] Key type .llcrypt registered [ 7910.961886] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 7912.151909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7912.196182] alg: No test for adler32 (adler32-zlib) [ 7913.270679] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 7913.479702] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 7915.143938] Key type lgssc registered [ 7916.343150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7928.046553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7928.096782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7929.769967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7936.008897] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7940.070253] LustreError: 235679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7940.106804] LustreError: 235679:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7945.190398] LustreError: 235680:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7946.869278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7947.091971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7951.026831] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7954.551729] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7961.335459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7961.751137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7970.445926] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7976.125520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7981.069399] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7981.283906] Lustre: lustre-OST0001: new disk, initializing [ 7981.292955] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7981.297190] Lustre: 237987:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7981.392994] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7987.374567] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7990.478917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7990.491915] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7990.632243] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7999.587119] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8008.471850] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8029.153522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8029.173383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8029.188867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8031.720714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8031.731963] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8031.745286] Lustre: Skipped 2 previous similar messages [ 8031.770549] Lustre: Skipped 2 previous similar messages [ 8033.751789] Lustre: server umount lustre-MDT0000 complete [ 8036.836889] LustreError: 239007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8036.862100] LustreError: 239007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8038.393498] LustreError: 238869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594169 with bad export cookie 11683522897410965822 [ 8038.399846] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8038.403237] LustreError: 238869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8038.955405] Lustre: server umount lustre-MDT0001 complete [ 8054.196134] Lustre: server umount lustre-OST0000 complete [ 8068.508722] Lustre: server umount lustre-OST0001 complete [ 8094.091658] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 8105.310335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8105.748350] LustreError: 241095:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8105.887703] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8111.080300] LustreError: 241096:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8111.350115] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8121.314483] LustreError: 241095:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8121.338060] LustreError: 241095:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8121.685772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8122.200325] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8127.427031] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8130.923593] Lustre: 242239:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8138.400787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8138.696537] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8146.045979] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8146.927573] LustreError: 242593:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8151.060626] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49731 to 0x280000401:49760) [ 8156.768908] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8165.484735] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8174.735722] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8178.635457] Lustre: 244109:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8193.000702] Lustre: mylustre-MDT0000-lwp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8193.010650] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8193.024819] Lustre: Skipped 3 previous similar messages [ 8193.036329] Lustre: Skipped 4 previous similar messages [ 8198.120028] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8198.139807] Lustre: Skipped 3 previous similar messages [ 8203.249577] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8203.262775] Lustre: Skipped 3 previous similar messages [ 8204.255823] Lustre: mylustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8204.608413] Lustre: server umount mylustre-MDT0000 complete [ 8208.355819] LustreError: 242214:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8208.385991] LustreError: 242214:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 8209.291071] LustreError: 241072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594340 with bad export cookie 11683522897410968629 [ 8209.295661] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8209.305418] LustreError: 241072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8209.986769] Lustre: server umount mylustre-MDT0001 complete [ 8226.119530] Lustre: server umount mylustre-OST0000 complete [ 8229.649261] Lustre: 234939:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788594345/real 1788594345] req@ffff88d6c27ce300 x1875476789493376/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788594361 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8229.691848] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8230.882592] Lustre: 234937:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788594346/real 1788594346] req@ffff88d7cf4a7480 x1875476789493632/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788594362 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8232.370930] Lustre: server umount mylustre-OST0001 complete [ 8260.340661] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 8273.239919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8273.821333] LustreError: 246422:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8273.889255] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8273.897359] Lustre: Skipped 1 previous similar message [ 8279.338957] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8289.205116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8295.148701] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8298.611946] Lustre: 247564:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8306.215255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8313.209485] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8318.943261] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49731 to 0x280000401:49792) [ 8322.480854] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8328.190508] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 8330.204857] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8338.559410] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8342.968959] Lustre: 249438:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8351.208582] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8351.216689] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8351.240161] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 8352.771511] Lustre: server umount tfs-MDT0000 complete [ 8356.981382] LustreError: 246399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594488 with bad export cookie 11683522897410970295 [ 8356.991021] LustreError: 246399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8356.992162] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8357.499499] Lustre: server umount tfs-MDT0001 complete [ 8372.521701] Lustre: server umount tfs-OST0000 complete [ 8374.431140] Lustre: 234939:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788594490/real 1788594490] req@ffff88d7cf4a5180 x1875476789545344/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788594506 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8374.473194] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8374.495266] Lustre: Skipped 3 previous similar messages [ 8377.189982] Lustre: server umount tfs-OST0001 complete [ 8402.074514] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 8414.119195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8414.943029] LustreError: 251686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8414.966250] LustreError: 251686:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 8415.054927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8415.074349] Lustre: Skipped 3 previous similar messages [ 8420.462745] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8431.056983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8436.918728] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8440.102707] Lustre: 252829:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8448.338988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8456.331284] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8461.057087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49731 to 0x280000401:49824) [ 8466.385170] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8472.068224] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 8474.345608] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8483.247017] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8487.231821] Lustre: 254699:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8495.337123] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 03:50:25 (1788594625) [ 8512.993651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8513.018285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8513.019764] Lustre: Skipped 2 previous similar messages [ 8513.031206] Lustre: Skipped 2 previous similar messages [ 8526.303362] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8526.544546] Lustre: server umount lustre-MDT0000 complete [ 8530.774649] LustreError: 251662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594662 with bad export cookie 11683522897410971954 [ 8530.784303] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8530.784405] LustreError: 251662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8545.247657] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8545.636271] Lustre: server umount lustre-MDT0001 complete [ 8556.399679] Lustre: server umount lustre-OST0000 complete [ 8567.231786] Lustre: server umount lustre-OST0001 complete [ 8574.670910] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 8584.893546] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 8635.535627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8636.023930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8636.096606] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8636.360922] Lustre: lustre-MDT0000: new disk, initializing [ 8636.521522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 8636.530855] Lustre: Skipped 3 previous similar messages [ 8636.566891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8642.972602] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8656.670720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8656.786690] Lustre: 257897:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8656.877799] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8656.882386] Lustre: Skipped 1 previous similar message [ 8657.006714] Lustre: lustre-MDT0001: new disk, initializing [ 8657.178781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8657.185570] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8662.713431] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8675.828549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8676.248175] Lustre: lustre-OST0000: new disk, initializing [ 8676.257506] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8676.276417] Lustre: 259480:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8677.512877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8677.527643] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8677.736049] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8684.236969] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8695.896551] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8696.122318] Lustre: lustre-OST0001: new disk, initializing [ 8696.125157] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8696.129664] Lustre: 260351:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8697.848991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8697.878327] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8698.130083] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8704.426561] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8718.130632] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 03:54:07 (1788594847) [ 8734.237887] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 03:54:24 (1788594864) [ 8754.659120] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8754.677460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8754.699541] Lustre: Skipped 3 previous similar messages [ 8754.704443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8754.711789] Lustre: Skipped 20 previous similar messages [ 8756.541043] Lustre: server umount lustre-OST0000 complete [ 8759.794083] LustreError: 259562:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8759.827771] LustreError: 259562:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [ 8764.910845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8764.918026] LustreError: Skipped 1 previous similar message [ 8767.944455] Lustre: server umount lustre-MDT0000 complete [ 8772.978992] LustreError: 257889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594904 with bad export cookie 11683522897410973515 [ 8772.987703] LustreError: 257889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8772.996100] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8779.645281] Lustre: server umount lustre-MDT0001 complete [ 8792.422489] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 8869.209944] Lustre: server umount lustre-OST0001 complete [ 8877.210187] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 8894.116630] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 8939.637492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8940.103055] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8940.174618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8940.352874] Lustre: lustre-MDT0000: new disk, initializing [ 8940.502311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8940.507695] Lustre: Skipped 3 previous similar messages [ 8940.563991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8946.941817] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8959.484225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8959.628382] Lustre: 265210:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8959.677473] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8959.681275] Lustre: Skipped 1 previous similar message [ 8959.766351] Lustre: lustre-MDT0001: new disk, initializing [ 8959.988888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8959.997118] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8965.143180] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8976.442541] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8982.834776] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8990.541310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8990.877737] Lustre: lustre-OST0000: new disk, initializing [ 8990.882559] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8990.890525] Lustre: 267176:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8992.047667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8992.063708] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8992.291078] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8998.317373] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9010.223645] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9017.824819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9017.827917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9017.847684] LustreError: Skipped 1 previous similar message [ 9017.854742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9017.871095] Lustre: Skipped 6 previous similar messages [ 9017.891864] Lustre: Skipped 6 previous similar messages [ 9023.972241] Lustre: server umount lustre-OST0000 complete [ 9031.649248] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9034.864691] Lustre: server umount lustre-MDT0000 complete [ 9039.289103] LustreError: 265201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595170 with bad export cookie 11683522897410974978 [ 9039.309887] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9039.707241] Lustre: server umount lustre-MDT0001 complete [ 9051.920725] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 9055.741212] Key type lgssc unregistered [ 9056.052708] LNet: 268936:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9056.061361] LNetError: 268936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9056.087170] LNet: Removed LNI 192.168.202.159@tcp [ 9056.960234] Key type .llcrypt unregistered [ 9056.964369] Key type ._llcrypt unregistered [ 9085.803979] Key type ._llcrypt registered [ 9085.805616] Key type .llcrypt registered [ 9085.929430] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 9087.201541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9087.214175] alg: No test for adler32 (adler32-zlib) [ 9088.471869] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 9088.710401] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 9090.487152] Key type lgssc registered [ 9091.771093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9099.861867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9099.884464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9101.553501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9106.690587] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9112.033427] LustreError: 270395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9112.051323] LustreError: 270395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9115.589969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9116.115862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9120.868579] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9129.622270] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9136.609351] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9143.264904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9143.511394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9146.610791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 9149.650449] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9159.997228] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9161.109471] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 9166.310899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9166.315615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9166.329044] Lustre: Skipped 1 previous similar message [ 9166.348646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9170.542903] Lustre: server umount lustre-OST0000 complete [ 9176.546567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9176.576678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9176.586958] Lustre: Skipped 2 previous similar messages [ 9181.669351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9181.684526] Lustre: Skipped 1 previous similar message [ 9182.136189] Lustre: server umount lustre-MDT0000 complete [ 9186.792660] LustreError: 270395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9186.827787] LustreError: 270395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9186.920067] LustreError: 270377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595318 with bad export cookie 10074382211937245825 [ 9186.929922] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9186.933862] LustreError: 270377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9187.341764] Lustre: server umount lustre-MDT0001 complete [ 9199.876212] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 9203.185563] Key type lgssc unregistered [ 9203.663714] LNet: 273990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9203.679575] LNetError: 273990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9203.708374] LNet: Removed LNI 192.168.202.159@tcp [ 9204.701292] Key type .llcrypt unregistered [ 9204.703066] Key type ._llcrypt unregistered [ 9224.944652] Key type ._llcrypt registered [ 9224.951606] Key type .llcrypt registered [ 9225.073155] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 9226.565315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9226.615995] alg: No test for adler32 (adler32-zlib) [ 9227.781919] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 9228.085586] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 9229.767191] Key type lgssc registered [ 9231.215502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9241.314912] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9241.329221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9256.867217] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9261.983726] LustreError: 275192:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 4: rc = -110 [ 9288.735545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9295.671842] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9305.156939] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9307.780701] Lustre: server umount lustre-OST0000 complete [ 9327.764483] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 9330.726473] Key type lgssc unregistered [ 9331.233158] LNet: 276821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9331.255726] LNetError: 276821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9331.275450] LNet: Removed LNI 192.168.202.159@tcp [ 9332.302680] Key type .llcrypt unregistered [ 9332.306415] Key type ._llcrypt unregistered [ 9378.527158] Key type ._llcrypt registered [ 9378.530732] Key type .llcrypt registered [ 9378.694953] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 9379.965088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9380.023370] alg: No test for adler32 (adler32-zlib) [ 9381.174647] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 9381.389921] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 9383.042969] Key type lgssc registered [ 9384.076021] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9397.114961] loop0: detected capacity change from 0 to 204800000 [ 9397.294380] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9397.326785] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9397.687927] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9399.022362] loop0: detected capacity change from 0 to 204800000 [ 9399.246350] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9399.260079] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9399.504120] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9400.791375] loop0: detected capacity change from 0 to 204800000 [ 9401.049551] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9401.101407] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9401.466457] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9403.108473] loop0: detected capacity change from 0 to 204800000 [ 9403.442420] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9403.470187] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9403.737804] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9405.077605] loop0: detected capacity change from 0 to 204800000 [ 9405.097592] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9411.531514] loop0: detected capacity change from 0 to 204800000 [ 9411.590795] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9417.454195] loop0: detected capacity change from 0 to 204800000 [ 9417.537488] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9423.225359] loop0: detected capacity change from 0 to 204800000 [ 9423.259315] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9427.732933] loop0: detected capacity change from 0 to 204800000 [ 9427.885951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9427.958353] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9428.038518] Lustre: 280237:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9428.044550] Lustre: 280237:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 9428.051535] Lustre: 280237:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9428.058737] Lustre: 280237:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9428.066433] Lustre: 280237:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 9428.074374] Lustre: 280237:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9428.081513] Lustre: 280237:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 9428.090777] Lustre: 280237:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 9428.102209] Lustre: 280237:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 9428.117258] Lustre: 280237:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 9428.130233] Lustre: 280237:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 9428.136606] Lustre: 280237:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 9428.148802] Lustre: 280237:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 9428.156944] Lustre: 280237:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 9434.783915] Lustre: server umount MGS complete [ 9435.802991] loop0: detected capacity change from 0 to 204800000 [ 9435.904280] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9436.767251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9436.798933] Lustre: lustre-MDT0000: Aborting client recovery [ 9437.921764] loop1: detected capacity change from 0 to 204800000 [ 9438.145104] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9438.248855] Lustre: 280671:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9438.268270] Lustre: 280671:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 9438.282471] Lustre: 280671:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 9438.294171] Lustre: 280671:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9438.317480] Lustre: 280671:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9438.330886] Lustre: 280671:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9438.351961] Lustre: 280671:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 9438.385222] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9438.517477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9438.582660] Lustre: lustre-MDT0001: Aborting client recovery [ 9439.724378] loop2: detected capacity change from 0 to 204800000 [ 9440.030880] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9440.161484] Lustre: 280750:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9440.188719] Lustre: 280750:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9440.213525] Lustre: 280750:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9440.228616] Lustre: 280750:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9440.239939] Lustre: 280750:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9440.250476] Lustre: 280750:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9440.261596] Lustre: 280750:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9440.427797] Lustre: Found index 0 for lustre-OST0000, updating log [ 9440.484905] Lustre: 280770:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9440.613346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9440.672884] Lustre: lustre-OST0000: Aborting recovery [ 9441.786796] loop3: detected capacity change from 0 to 204800000 [ 9441.962138] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9442.074970] Lustre: 280837:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9442.087506] Lustre: 280837:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9442.105598] Lustre: 280837:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9442.114490] Lustre: 280837:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9442.124834] Lustre: 280837:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9442.134934] Lustre: 280837:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9442.147984] Lustre: 280837:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9442.183506] Lustre: Found index 1 for lustre-OST0001, updating log [ 9442.221675] Lustre: 280847:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9442.377830] Lustre: lustre-OST0001: Aborting recovery [ 9446.471544] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9446.506119] LustreError: 280615:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 9446.533946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9446.559241] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 9446.597432] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 9448.489598] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 9448.493198] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 9451.278769] LustreError: 280582:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 3184us: rc = -2 [ 9451.790632] LustreError: 280579:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 777us: rc = -2 [ 9451.813792] LustreError: 280579:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 18 previous similar messages [ 9468.908520] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9468.911740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9468.922056] Lustre: Skipped 2 previous similar messages [ 9468.944516] Lustre: Skipped 3 previous similar messages [ 9471.819776] Lustre: server umount lustre-MDT0000 complete [ 9472.999071] LustreError: 280564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595604 with bad export cookie 11104703871889985656 [ 9473.004552] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9473.010457] LustreError: 280564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9474.021285] LustreError: 280580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9474.025819] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9474.048551] LustreError: 280580:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9474.065651] Lustre: Skipped 1 previous similar message [ 9474.080622] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9477.024886] LustreError: 280580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9477.031535] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9477.042651] LustreError: 280580:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9477.060859] Lustre: Skipped 2 previous similar messages [ 9479.659484] Lustre: server umount lustre-MDT0001 complete [ 9486.872541] Lustre: server umount lustre-OST0000 complete [ 9494.314887] Lustre: server umount lustre-OST0001 complete [ 9503.317678] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 04:07:13 (1788595633) [ 9535.631642] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 9547.135993] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 9590.105382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9590.463050] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9590.543278] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9590.723434] Lustre: lustre-MDT0000: new disk, initializing [ 9590.900935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9590.911454] Lustre: Skipped 1 previous similar message [ 9590.951112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9596.481537] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9608.689214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9608.837115] Lustre: 284915:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9608.912921] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9608.920854] Lustre: Skipped 1 previous similar message [ 9609.018402] Lustre: lustre-MDT0001: new disk, initializing [ 9609.176218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9609.220102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9609.236833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9614.321291] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9624.621963] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9632.383595] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9641.602697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9641.749816] Lustre: lustre-OST0000: new disk, initializing [ 9641.751572] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9641.754267] Lustre: 286886:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9641.800225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9650.767246] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9651.809682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9651.825247] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9652.041664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9663.094952] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9668.955831] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9679.069463] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9685.747514] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9729.000713] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9729.005383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9729.020078] Lustre: Skipped 2 previous similar messages [ 9729.033829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9734.137296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9734.151143] Lustre: Skipped 2 previous similar messages [ 9734.602257] Lustre: server umount lustre-OST0000 complete [ 9739.233262] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9739.254437] Lustre: Skipped 1 previous similar message [ 9744.363127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9744.380766] Lustre: Skipped 3 previous similar messages [ 9745.363339] Lustre: server umount lustre-MDT0000 complete [ 9749.472488] LustreError: 284921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9749.494314] LustreError: 284921:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9750.105961] LustreError: 285711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595881 with bad export cookie 11104703871889988421 [ 9750.110334] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9750.124630] LustreError: 285711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9750.578255] Lustre: server umount lustre-MDT0001 complete [ 9761.703648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9767.188509] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9773.578631] Lustre: server umount MGS complete [ 9785.981645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9786.331843] LustreError: 290703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9786.358924] LustreError: 290716:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9786.420277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9791.459376] LustreError: 290704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9791.658334] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9800.516444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9800.686194] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9806.836993] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9815.609444] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9821.735585] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9830.847171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9831.351732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9831.364243] Lustre: Skipped 1 previous similar message [ 9839.009047] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9847.964915] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9891.815379] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9891.847871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9891.852545] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9894.917548] Lustre: server umount lustre-OST0000 complete [ 9902.051179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9902.072215] Lustre: Skipped 1 previous similar message [ 9905.669332] Lustre: server umount lustre-MDT0000 complete [ 9907.173044] LustreError: 292171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9907.224979] LustreError: 292171:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9909.629161] LustreError: 290685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788596041 with bad export cookie 11104703871889990255 [ 9909.631899] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9909.639137] LustreError: 290685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9909.999077] Lustre: server umount lustre-MDT0001 complete [ 9924.751980] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 9928.793694] Key type lgssc unregistered [ 9929.293246] LNet: 294670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9929.313787] LNetError: 294670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9930.360615] LNet: Removed LNI 192.168.202.159@tcp [ 9931.677092] Key type .llcrypt unregistered [ 9931.679974] Key type ._llcrypt unregistered [ 9976.285558] Key type ._llcrypt registered [ 9976.287855] Key type .llcrypt registered [ 9976.398399] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 9993.471338] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 9995.259961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9995.311124] alg: No test for adler32 (adler32-zlib) [ 9996.467454] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 9996.726733] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 9998.391167] Key type lgssc registered [ 9999.804772] Lustre: Echo OBD driver; http://www.lustre.org/ [10039.740907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10039.782062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10041.113231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10041.175589] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10041.369812] Lustre: lustre-MDT0000: new disk, initializing [10041.569946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10041.604339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10046.497724] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10058.871658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10059.060954] Lustre: 298626:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10059.131503] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10059.134584] Lustre: Skipped 1 previous similar message [10059.233613] Lustre: lustre-MDT0001: new disk, initializing [10059.361078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10059.428142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10059.449337] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10065.320309] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10077.435896] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10084.243541] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10092.426529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10092.825608] Lustre: lustre-OST0000: new disk, initializing [10092.828822] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10092.834297] Lustre: 300594:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10092.945401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10100.321379] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10101.395788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10101.408709] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10101.585498] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10111.841396] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10119.048595] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [10130.453761] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [10135.380993] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [10193.889620] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10193.893752] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10193.903976] LustreError: Skipped 1 previous similar message [10193.904884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10193.956871] Lustre: Skipped 1 previous similar message [10195.375650] Lustre: server umount lustre-OST0000 complete [10202.592523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10202.599788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10202.618220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10202.623608] Lustre: Skipped 1 previous similar message [10204.130541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10204.134963] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10206.798272] Lustre: server umount lustre-MDT0000 complete [10211.316492] LustreError: 298617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788596342 with bad export cookie 15630501046991065287 [10211.318510] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10211.323989] LustreError: 298617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10211.660564] Lustre: server umount lustre-MDT0001 complete [10221.351584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10227.349243] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10232.901448] Lustre: server umount MGS complete [10245.958539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10246.432785] LustreError: 304368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10246.456513] LustreError: 304368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10246.496351] LustreError: 304381:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [10246.609313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10251.381141] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10251.746481] LustreError: 304369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10256.867918] LustreError: 304368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10261.548308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10261.958778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10267.487933] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10275.599413] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10282.070908] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10289.629926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10290.067515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10298.452529] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10307.418519] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10343.405677] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10343.416131] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10343.421479] Lustre: Skipped 1 previous similar message [10343.423184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10343.430587] LustreError: Skipped 1 previous similar message [10343.446914] Lustre: Skipped 2 previous similar messages [10346.331178] Lustre: server umount lustre-OST0000 complete [10353.633673] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10353.649520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10353.658571] Lustre: Skipped 1 previous similar message [10363.872768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10363.878936] Lustre: Skipped 3 previous similar messages [10364.383118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10364.716286] Lustre: server umount lustre-MDT0000 complete [10368.994315] LustreError: 306209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10369.026804] LustreError: 306209:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10370.798183] LustreError: 304349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788596502 with bad export cookie 15630501046991067128 [10370.801505] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10370.821748] LustreError: 304349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10371.333795] Lustre: server umount lustre-MDT0001 complete [10385.564466] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [10389.552555] Key type lgssc unregistered [10389.998880] LNet: 308316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10390.009701] LNetError: 308316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10390.042362] LNet: Removed LNI 192.168.202.159@tcp [10391.160987] Key type .llcrypt unregistered [10391.164694] Key type ._llcrypt unregistered [10415.189854] Key type ._llcrypt registered [10415.192809] Key type .llcrypt registered [10415.366092] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [10433.041796] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [10434.458702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10434.525388] alg: No test for adler32 (adler32-zlib) [10435.658812] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [10435.844512] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [10437.495281] Key type lgssc registered [10438.904395] Lustre: Echo OBD driver; http://www.lustre.org/ [10466.466503] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [10482.508314] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10482.539438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10483.853695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10483.939774] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10484.075471] Lustre: lustre-MDT0000: new disk, initializing [10484.204213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10484.229330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10489.153403] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10500.534346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10500.769389] Lustre: lustre-OST0000: new disk, initializing [10500.772276] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10500.775866] Lustre: Skipped 1 previous similar message [10500.779118] Lustre: 312379:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10500.867651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10507.901964] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10512.878612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10512.905544] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10512.942310] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11458.054437] Lustre: server umount lustre-MDT0000 complete [11462.587517] LustreError: 311566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788597594 with bad export cookie 8220570266887170629 [11462.597201] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11473.024108] Lustre: server umount lustre-OST0000 complete [11503.440318] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [11513.773101] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [11553.897644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11554.093632] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11554.144518] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11554.225993] Lustre: lustre-MDT0000: new disk, initializing [11554.298706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11554.328157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11558.855360] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11570.216377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11570.379950] Lustre: 316733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11570.430532] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11570.433415] Lustre: Skipped 1 previous similar message [11570.485249] Lustre: lustre-MDT0001: new disk, initializing [11570.554580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11570.598392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11570.603824] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11574.997559] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11587.409844] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11595.268783] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11604.576135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11605.087499] Lustre: lustre-OST0000: new disk, initializing [11605.091313] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11605.102356] Lustre: 318704:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11605.223332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11611.760663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11611.783801] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11611.969040] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11612.390945] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11623.328966] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11630.061753] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11630.411944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11636.462877] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11636.848326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11642.850824] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11642.864053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11642.864354] LustreError: Skipped 1 previous similar message [11642.874029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11642.891299] Lustre: Skipped 1 previous similar message [11642.921516] Lustre: Skipped 1 previous similar message [11645.959649] Lustre: server umount lustre-OST0000 complete [11652.576190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11652.580286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11652.589616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11656.504818] Lustre: server umount lustre-MDT0000 complete [11661.141693] LustreError: 318703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788597792 with bad export cookie 8220570266890111448 [11661.147594] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11661.148530] LustreError: 318703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11661.679133] Lustre: server umount lustre-MDT0001 complete [11666.061372] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [11668.029820] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 04:43:18 (1788597798) [11677.565326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11678.152967] LustreError: 321019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11678.181092] LustreError: 321019:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11678.243493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11683.300049] LustreError: 321020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11683.954377] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11688.420789] LustreError: 321019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11693.540830] LustreError: 321020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11694.781095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11695.268265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11700.888668] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11709.513714] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11716.768862] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11725.658949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11726.066710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11734.968410] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11745.614547] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11754.861238] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11755.144546] Lustre: lustre-OST0001: new disk, initializing [11755.148250] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11755.153114] Lustre: 323856:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11755.200725] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11762.571474] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11763.350647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11763.370220] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11763.456055] LustreError: 324243:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11764.520048] LustreError: 324248:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11764.545641] LustreError: 324248:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [11765.603990] LustreError: 324248:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11765.635391] LustreError: 324248:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [11767.717028] LustreError: 324243:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11767.745209] LustreError: 324243:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [11771.938398] LustreError: 324248:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11771.960964] LustreError: 324248:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [11774.237941] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11774.495602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11780.153436] LustreError: 324243:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11780.162803] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff94e104761800.ost_server_uuid 50 [11780.175637] LustreError: 324243:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 14 previous similar messages [11782.253739] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff94e104761800.ost_server_uuid in FULL state after 0 sec [11790.512519] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff94e104761800.ost_server_uuid 50 [11792.226775] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff94e104761800.ost_server_uuid in IDLE state after 0 sec [11794.706388] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11809.767119] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11809.792615] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11809.794309] Lustre: Skipped 2 previous similar messages [11809.811867] Lustre: Skipped 3 previous similar messages [11814.874718] Lustre: server umount lustre-OST0001 complete [11814.893494] LustreError: 322840:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11814.913644] LustreError: 322840:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11822.568628] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11822.580741] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11822.597465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11822.609775] Lustre: Skipped 1 previous similar message [11824.608769] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11825.140538] LustreError: 323558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11825.161950] LustreError: 323558:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11827.743900] Lustre: server umount lustre-OST0000 complete [11833.827719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11833.832147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11833.876891] Lustre: Skipped 1 previous similar message [11833.893822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11833.910965] Lustre: Skipped 3 previous similar messages [11837.708807] Lustre: server umount lustre-MDT0000 complete [11841.568721] LustreError: 323855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788597973 with bad export cookie 8220570266890112330 [11841.576054] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11841.580209] LustreError: 323855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11841.853623] Lustre: server umount lustre-MDT0001 complete [11853.200270] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [11855.601561] Key type lgssc unregistered [11855.820464] LNet: 326546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11855.827930] LNetError: 326546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11855.849730] LNet: Removed LNI 192.168.202.159@tcp [11856.872575] Key type .llcrypt unregistered [11856.873805] Key type ._llcrypt unregistered [11884.117053] Key type ._llcrypt registered [11884.119504] Key type .llcrypt registered [11884.297917] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [11886.570265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11886.633373] alg: No test for adler32 (adler32-zlib) [11887.902126] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [11888.176668] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [11889.832335] Key type lgssc registered [11891.145937] Lustre: Echo OBD driver; http://www.lustre.org/ [11901.741567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11901.786494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11903.351053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11907.654094] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11912.681973] LustreError: 327962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11912.707209] LustreError: 327962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11914.605360] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11917.796142] LustreError: 327963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11921.585987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11921.946821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11926.450700] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11939.083808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11939.702201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11946.882351] LustreError: 329642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11947.969378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [11948.969334] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11953.130895] LustreError: 329641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11953.185041] LustreError: 329641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11957.969831] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11958.249689] LustreError: 329641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11958.271804] LustreError: 329641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11966.123711] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11966.490844] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11971.589288] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [11974.217388] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11982.333664] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11990.762155] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [11991.068338] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12104.681522] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12104.686537] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12104.697948] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12104.712458] Lustre: Skipped 1 previous similar message [12104.755870] Lustre: Skipped 1 previous similar message [12106.523428] Lustre: server umount lustre-OST0001 complete [12109.810516] LustreError: 329642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12109.853066] LustreError: 329642:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12114.913336] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12114.934167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12114.938314] Lustre: Skipped 1 previous similar message [12120.041818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12120.044418] Lustre: Skipped 1 previous similar message [12120.382350] Lustre: server umount lustre-OST0000 complete [12125.152219] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12125.154356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12125.179789] Lustre: Skipped 2 previous similar messages [12125.195530] Lustre: Skipped 1 previous similar message [12130.272439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12130.282851] Lustre: Skipped 1 previous similar message [12130.870511] Lustre: server umount lustre-MDT0000 complete [12134.842028] LustreError: 327943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788598266 with bad export cookie 2004973013212493036 [12134.844868] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12134.867199] LustreError: 327943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12135.509494] Lustre: server umount lustre-MDT0001 complete [12147.868390] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [12151.025176] Key type lgssc unregistered [12151.400654] LNet: 332907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12151.410709] LNetError: 332907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12151.427264] LNet: Removed LNI 192.168.202.159@tcp [12152.336191] Key type .llcrypt unregistered [12152.338745] Key type ._llcrypt unregistered [12197.360765] Key type ._llcrypt registered [12197.364143] Key type .llcrypt registered [12197.478873] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [12198.028097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12198.042197] alg: No test for adler32 (adler32-zlib) [12199.230857] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [12199.444867] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [12201.223189] Key type lgssc registered [12202.602775] Lustre: Echo OBD driver; http://www.lustre.org/ [12212.772886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12212.812510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12214.524591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12219.710367] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12224.996765] LustreError: 335194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12225.029483] LustreError: 335194:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12229.946339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12230.113776] LustreError: 335193:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12230.629217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12235.801986] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12239.655142] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12247.743087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12248.001307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12254.077505] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12256.188182] LustreError: 336683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12260.286502] LustreError: 337146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12260.299423] LustreError: 337146:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12260.330346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [12264.094710] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12264.335644] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12269.552128] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [12271.775947] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12281.964617] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12288.882606] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [12305.386698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12305.399881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12305.410567] Lustre: Skipped 3 previous similar messages [12305.435935] Lustre: Skipped 3 previous similar messages [12309.358913] Lustre: server umount lustre-MDT0000 complete [12310.513229] LustreError: 335193:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12310.533953] LustreError: 335193:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12313.054784] LustreError: 336330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788598444 with bad export cookie 625656187412707940 [12313.076835] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12313.718931] Lustre: server umount lustre-MDT0001 complete [12329.216193] Lustre: server umount lustre-OST0000 complete [12332.128301] Lustre: 334458:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788598447/real 1788598447] req@ffff88d6d2631f80 x1875481283592832/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788598463 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12332.172407] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12333.691715] Lustre: server umount lustre-OST0001 complete [12352.717759] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [12364.578241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12365.121720] LustreError: 340464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12365.185792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12370.997055] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12380.643607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12381.318101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12386.527770] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12389.725128] Lustre: 341604:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [12397.912605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12398.286841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12405.493366] LustreError: 341956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12405.512988] LustreError: 341956:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12405.689739] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12409.588535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [12414.547886] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12420.095567] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [12425.090740] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12435.806754] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12439.812376] Lustre: 343478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12447.716423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12447.727764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12447.741946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12449.194768] Lustre: server umount lustre-MDT0000 complete [12450.787873] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12450.794247] LustreError: 340464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12450.800528] Lustre: Skipped 1 previous similar message [12450.825979] LustreError: 340464:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12453.178528] LustreError: 340444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788598584 with bad export cookie 625656187412709641 [12453.190017] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12453.203002] LustreError: 340444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12453.695920] Lustre: server umount lustre-MDT0001 complete [12467.705932] Lustre: server umount lustre-OST0000 complete [12471.455422] Lustre: 334456:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788598587/real 1788598587] req@ffff88d6ce75ca80 x1875481283648768/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788598603 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12471.469509] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12471.486106] Lustre: Skipped 1 previous similar message [12471.677585] Lustre: server umount lustre-OST0001 complete [12476.570853] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [12478.942995] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [12483.313845] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 04:56:51 (1788598611) [12511.090243] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [12520.302498] loop0: detected capacity change from 0 to 1099511627776 [12521.052816] SGI XFS with ACLs, security attributes, quota, no debug enabled [12521.060737] *** VALIDATE xfs *** [12521.122466] XFS (loop0): Mounting V5 Filesystem [12521.185290] XFS (loop0): Ending clean mount [12522.662627] loop1: detected capacity change from 0 to 18691697672192 [12645.568777] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [12661.093136] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [12707.982410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12708.401468] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12708.463865] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12708.541444] Lustre: lustre-MDT0000: new disk, initializing [12708.633852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12708.643518] Lustre: Skipped 1 previous similar message [12708.666577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12713.717758] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12726.086082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12726.243718] Lustre: 348645:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12726.286786] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12726.296826] Lustre: Skipped 1 previous similar message [12726.422516] Lustre: lustre-MDT0001: new disk, initializing [12726.582784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12726.595601] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12730.710693] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12741.071286] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12747.545045] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12755.147270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12755.372563] Lustre: lustre-OST0000: new disk, initializing [12755.376897] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12755.384613] Lustre: 350624:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12757.367302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12757.383501] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12757.579673] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12760.977792] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12770.201468] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12776.458230] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12776.862783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12784.144899] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12784.387668] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12788.202738] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12788.203968] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12788.226134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12788.235293] LustreError: Skipped 1 previous similar message [12793.318710] LustreError: 350608:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12793.352763] LustreError: 350608:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12793.455762] Lustre: server umount lustre-OST0000 complete [12803.559759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12803.560490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12803.570182] Lustre: Skipped 2 previous similar messages [12805.182885] Lustre: server umount lustre-MDT0000 complete [12810.360444] LustreError: 348637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788598941 with bad export cookie 625656187412711195 [12810.365703] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12810.375474] LustreError: 348637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12810.790934] Lustre: server umount lustre-MDT0001 complete [12819.949299] XFS (loop0): Unmounting Filesystem [12825.735969] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 05:02:35 (1788598955) [12833.443312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12834.149128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12834.158182] Lustre: Skipped 2 previous similar messages [12839.307821] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12848.681674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12854.523396] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12864.163951] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12871.263922] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12878.887886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12886.580274] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [12895.355757] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12919.274245] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12919.285028] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12919.291888] LustreError: Skipped 1 previous similar message [12919.302733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12919.305681] Lustre: Skipped 4 previous similar messages [12920.835953] Lustre: server umount lustre-OST0000 complete [12925.920716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12929.505884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12929.511685] Lustre: Skipped 2 previous similar messages [12939.231756] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12939.468828] Lustre: server umount lustre-MDT0000 complete [12943.766664] LustreError: 353921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599075 with bad export cookie 625656187412712070 [12943.775721] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12943.790350] LustreError: 353921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12944.213991] Lustre: server umount lustre-MDT0001 complete [12956.597767] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [12959.810582] Key type lgssc unregistered [12960.128080] LNet: 356839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12960.135538] LNetError: 356839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12960.149183] LNet: Removed LNI 192.168.202.159@tcp [12961.079189] Key type .llcrypt unregistered [12961.083037] Key type ._llcrypt unregistered [12991.139921] Key type ._llcrypt registered [12991.145463] Key type .llcrypt registered [12991.345516] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [12992.251558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12992.264272] alg: No test for adler32 (adler32-zlib) [12993.481633] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [12993.866398] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [12995.623201] Key type lgssc registered [12996.925189] Lustre: Echo OBD driver; http://www.lustre.org/ [13005.891411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13005.954884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13007.608383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13012.743581] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [13018.095499] LustreError: 358407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13018.121163] LustreError: 358407:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13022.067511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13022.448545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13027.452417] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [13036.063838] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13043.492962] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13051.120803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13051.602505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13058.355504] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [13066.564806] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13077.473701] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13077.488457] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13077.495306] Lustre: Skipped 1 previous similar message [13079.821821] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.59@tcp (stopping) [13081.962393] Lustre: server umount lustre-MDT0001 complete [13082.605385] LustreError: 358407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13082.630845] LustreError: 358407:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13084.888621] LustreError: 358402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13087.716369] LustreError: 358401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13087.744347] LustreError: 358401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13092.836364] LustreError: 358401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13092.858085] LustreError: 358401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13094.464707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13094.620704] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13094.623239] Lustre: Found index 1 for lustre-MDT0001, updating log [13094.686472] Lustre: 358394:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13095.223314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13098.659969] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [13100.530053] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13103.610610] LustreError: 362006:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13103.636046] LustreError: 362006:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13103.657277] LustreError: 358398:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13103.773629] LustreError: 362009:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.159@tcp: cfg command failed: rc = -17 [13103.786795] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [13103.786795] [13407.462561] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.59@tcp (stopping) [13407.713438] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13407.727457] Lustre: Skipped 1 previous similar message [13411.625405] Lustre: server umount lustre-MDT0001 complete [13412.574530] LustreError: 358403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13422.443456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13422.558880] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13422.564433] Lustre: Found index 1 for lustre-MDT0001, updating log [13422.623731] Lustre: 358394:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13422.929464] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13427.827269] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [13428.199782] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13429.763995] LustreError: 370427:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13429.773573] LustreError: 370427:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13429.782839] LustreError: 358398:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13429.791607] LustreError: 358398:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13429.886953] LustreError: 370438:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [13429.892036] LustreError: 370438:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [13429.901557] LustreError: 370438:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.159@tcp: cfg command failed: rc = -17 [13429.909849] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.202.159@tcp [13429.909849] [13735.180042] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.59@tcp (stopping) [13735.206590] Lustre: Skipped 2 previous similar messages [13735.395967] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13735.399863] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13735.414971] Lustre: Skipped 1 previous similar message [13740.254857] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.59@tcp (stopping) [13740.261421] Lustre: Skipped 2 previous similar messages [13749.215322] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13749.520490] Lustre: server umount lustre-MDT0001 complete [13750.490771] LustreError: 359405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13750.519584] LustreError: 359405:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13763.154270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13763.296638] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13763.301496] Lustre: Found index 1 for lustre-MDT0001, updating log [13763.362689] Lustre: 358396:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13763.674316] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13768.698977] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13768.757262] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [13771.269046] LustreError: 378663:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13771.288469] LustreError: 378663:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13771.299705] LustreError: 358398:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13771.305699] LustreError: 358398:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13771.416844] LustreError: 378673:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [13771.430310] LustreError: 378673:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.159@tcp: cfg command failed: rc = -114 [13771.444272] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [13771.444272] [14096.362978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14096.369725] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14096.377030] Lustre: Skipped 1 previous similar message [14096.379447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14096.434144] Lustre: Skipped 7 previous similar messages [14099.439815] Lustre: server umount lustre-MDT0000 complete [14101.475302] LustreError: 364186:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14101.490590] LustreError: 364186:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [14103.527808] LustreError: 358388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788600235 with bad export cookie 5498120669199376027 [14103.541775] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14103.545610] LustreError: 358388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14106.593525] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14106.617686] Lustre: Skipped 2 previous similar messages [14110.063208] Lustre: server umount lustre-MDT0001 complete [14114.355991] Lustre: server umount lustre-OST0000 complete [14124.564516] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [14139.655141] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [14181.692602] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 05:25:11 (1788600311) [14190.016706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14190.308496] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14190.337995] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14190.424565] Lustre: lustre-MDT0000: new disk, initializing [14190.555909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14190.578355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14195.874822] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [14207.499685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14207.635838] Lustre: 389209:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14207.698637] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14207.700961] Lustre: Skipped 1 previous similar message [14207.753341] Lustre: lustre-MDT0001: new disk, initializing [14207.851286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14207.886279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14207.898604] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14213.996543] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [14224.298196] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14231.914657] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14240.206894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14240.703698] Lustre: lustre-OST0000: new disk, initializing [14240.712263] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14240.725657] Lustre: 391182:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14240.877741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14248.369469] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [14248.552181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14248.563716] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14248.795215] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14258.681143] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14267.881163] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [14267.886636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14267.902443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14267.913625] Lustre: Skipped 2 previous similar messages [14268.668394] Lustre: server umount lustre-MDT0000 complete [14269.409816] LustreError: 389221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14269.450352] LustreError: 389221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14272.525928] LustreError: 389200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788600404 with bad export cookie 5498120669199377980 [14272.532689] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14272.536657] LustreError: 389200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14272.853507] Lustre: server umount lustre-MDT0001 complete [14286.652685] Lustre: server umount lustre-OST0000 complete [14320.842453] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [14331.369700] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [14363.499949] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14039 sec ========= 05:28:13 (1788600493) [14365.283386] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:28:15 (1788600495) === [14368.512890] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:28:18 (1788600498) === [14407.498750] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [14410.332987] Key type lgssc unregistered [14410.743729] LNet: 397546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14410.757233] LNetError: 397546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14410.773820] LNet: Removed LNI 192.168.202.159@tcp [14411.662108] Key type .llcrypt unregistered [14411.664817] Key type ._llcrypt unregistered