[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 466157128 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003263] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.007690] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010140] LSM: Security Framework initializing [ 0.011042] Yama: becoming mindful. [ 0.012037] SELinux: Initializing. [ 0.013063] *** VALIDATE selinux *** [ 0.022332] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027636] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028138] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029095] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030118] *** VALIDATE tmpfs *** [ 0.031572] *** VALIDATE proc *** [ 0.033097] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.035243] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038031] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.041687] debug: unmapping init [mem 0xffffffffac259000-0xffffffffac260fff] [ 0.043167] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044687] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045024] ... version: 2 [ 0.046013] ... bit width: 48 [ 0.047013] ... generic registers: 4 [ 0.048012] ... value mask: 0000ffffffffffff [ 0.049014] ... max period: 00007fffffffffff [ 0.050015] ... fixed-purpose events: 3 [ 0.051013] ... event mask: 000000070000000f [ 0.052311] rcu: Hierarchical SRCU implementation. [ 0.054493] smp: Bringing up secondary CPUs ... [ 0.055575] x86: Booting SMP configuration: [ 0.056026] .... node #0, CPUs: #1 #2 #3 [ 0.059099] smp: Brought up 1 node, 4 CPUs [ 0.061014] smpboot: Max logical packages: 1 [ 0.062014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.241996] node 0 deferred pages initialised in 177ms [ 0.245116] devtmpfs: initialized [ 0.246251] x86/mm: Memory block size: 128MB [ 0.249863] gcov: version magic: 0x41383552 [ 0.252227] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.255123] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.258357] pinctrl core: initialized pinctrl subsystem [ 0.260203] [ 0.260590] ************************************************************* [ 0.263014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.265024] ** ** [ 0.267013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.269013] ** ** [ 0.271013] ** This means that this kernel is built to expose internal ** [ 0.273015] ** IOMMU data structures, which may compromise security on ** [ 0.275013] ** your system. ** [ 0.277013] ** ** [ 0.280013] ** If you see this message and you are not debugging the ** [ 0.282011] ** kernel, report this immediately to your vendor! ** [ 0.284012] ** ** [ 0.286014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.289012] ************************************************************* [ 0.291671] NET: Registered protocol family 16 [ 0.293450] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.296057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.298053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.301032] cpuidle: using governor menu [ 0.302776] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.305447] PCI: Using configuration type 1 for base access [ 0.307133] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.316106] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.318024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.321166] cryptd: max_cpu_qlen set to 1000 [ 0.325310] ACPI: Added _OSI(Module Device) [ 0.327019] ACPI: Added _OSI(Processor Device) [ 0.329013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.330013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.334464] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.340514] ACPI: Interpreter enabled [ 0.342060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.343011] ACPI: Using IOAPIC for interrupt routing [ 0.345110] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.349409] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.358719] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.360037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.362017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.365088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.370308] acpiphp: Slot [2] registered [ 0.371095] acpiphp: Slot [5] registered [ 0.373112] acpiphp: Slot [6] registered [ 0.374110] acpiphp: Slot [7] registered [ 0.375127] acpiphp: Slot [8] registered [ 0.377103] acpiphp: Slot [9] registered [ 0.378097] acpiphp: Slot [10] registered [ 0.380094] acpiphp: Slot [3] registered [ 0.381096] acpiphp: Slot [4] registered [ 0.382091] acpiphp: Slot [11] registered [ 0.384112] acpiphp: Slot [12] registered [ 0.385119] acpiphp: Slot [13] registered [ 0.387168] acpiphp: Slot [14] registered [ 0.388211] acpiphp: Slot [15] registered [ 0.390138] acpiphp: Slot [16] registered [ 0.391128] acpiphp: Slot [17] registered [ 0.393149] acpiphp: Slot [18] registered [ 0.394126] acpiphp: Slot [19] registered [ 0.396136] acpiphp: Slot [20] registered [ 0.397143] acpiphp: Slot [21] registered [ 0.399135] acpiphp: Slot [22] registered [ 0.400120] acpiphp: Slot [23] registered [ 0.402148] acpiphp: Slot [24] registered [ 0.403130] acpiphp: Slot [25] registered [ 0.405134] acpiphp: Slot [26] registered [ 0.406176] acpiphp: Slot [27] registered [ 0.408136] acpiphp: Slot [28] registered [ 0.409133] acpiphp: Slot [29] registered [ 0.411125] acpiphp: Slot [30] registered [ 0.412131] acpiphp: Slot [31] registered [ 0.414127] PCI host bridge to bus 0000:00 [ 0.415027] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.417045] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.419035] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.422036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.425039] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.427044] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.429216] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.432086] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.436168] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.445010] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.450065] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.452025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.454022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.456020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.459036] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.461656] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.464042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.466817] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.472015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.483016] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.489024] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.495208] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.500023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.505023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.521021] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.531000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.538018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.543013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.558014] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.568386] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.580016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.587016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.603026] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.613995] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.617000] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.623017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.635019] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.648015] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.654030] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.661016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.675022] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.684792] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.691015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.697017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.710025] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.721055] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.723528] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.726442] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.728376] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.730253] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.736203] iommu: Default domain type: Passthrough [ 0.738393] SCSI subsystem initialized [ 0.739102] ACPI: bus type USB registered [ 0.741103] usbcore: registered new interface driver usbfs [ 0.742059] usbcore: registered new interface driver hub [ 0.743075] usbcore: registered new device driver usb [ 0.745158] pps_core: LinuxPPS API ver. 1 registered [ 0.746016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.748053] PTP clock support registered [ 0.750131] EDAC MC: Ver: 3.0.0 [ 0.752040] PCI: Using ACPI for IRQ routing [ 0.753752] NetLabel: Initializing [ 0.755023] NetLabel: domain hash size = 128 [ 0.756007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.757071] NetLabel: unlabeled traffic allowed by default [ 0.759106] vgaarb: loaded [ 0.760393] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.762017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.770096] clocksource: Switched to clocksource kvm-clock [ 0.876656] VFS: Disk quotas dquot_6.6.0 [ 0.877901] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.879845] *** VALIDATE ramfs *** [ 0.880941] *** VALIDATE hugetlbfs *** [ 0.882474] pnp: PnP ACPI init [ 0.884818] pnp: PnP ACPI: found 6 devices [ 0.909996] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.913194] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.915142] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.917229] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.919428] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.921657] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.924663] NET: Registered protocol family 2 [ 0.927133] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.931487] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.934770] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.941157] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.944722] TCP: Hash tables configured (established 65536 bind 65536) [ 0.948039] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.951104] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.953790] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.956568] NET: Registered protocol family 1 [ 0.958861] RPC: Registered named UNIX socket transport module. [ 0.961020] RPC: Registered udp transport module. [ 0.962783] RPC: Registered tcp transport module. [ 0.964551] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.967124] NET: Registered protocol family 44 [ 0.969119] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.971043] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.972970] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.975515] PCI: CLS 0 bytes, default 64 [ 0.977846] Unpacking initramfs... [ 2.397122] debug: unmapping init [mem 0xffff9d01fcc54000-0xffff9d01fffbffff] [ 2.403615] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.405531] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.409523] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.922217] Initialise system trusted keyrings [ 2.924090] Key type blacklist registered [ 2.926995] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.935885] zbud: loaded [ 2.939086] *** VALIDATE nfs *** [ 2.940683] *** VALIDATE nfs4 *** [ 2.942494] pstore: using deflate compression [ 2.945995] Platform Keyring initialized [ 3.048440] NET: Registered protocol family 38 [ 3.050130] Key type asymmetric registered [ 3.051579] Asymmetric key parser 'x509' registered [ 3.053369] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.056333] io scheduler mq-deadline registered [ 3.058497] io scheduler kyber registered [ 3.060263] io scheduler bfq registered [ 3.062131] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.065562] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.068430] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.071475] ACPI: Power Button [PWRF] [ 3.160762] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.251391] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.431836] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.524044] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.716521] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.744275] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.775104] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.780327] Non-volatile memory driver v1.3 [ 3.781895] Linux agpgart interface v0.103 [ 3.814946] virtio_blk virtio1: [vda] 133848 512-byte logical blocks (68.5 MB/65.4 MiB) [ 3.817710] vda: detected capacity change from 0 to 68530176 [ 3.840607] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.843820] vdb: detected capacity change from 0 to 1073741824 [ 3.861893] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.865301] vdc: detected capacity change from 0 to 2621440000 [ 3.881485] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.884498] vdd: detected capacity change from 0 to 2621440000 [ 3.898436] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.901589] vde: detected capacity change from 0 to 4294967296 [ 3.919595] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.922691] vdf: detected capacity change from 0 to 4294967296 [ 3.930449] libphy: Fixed MDIO Bus: probed [ 3.945243] usbcore: registered new interface driver usbserial_generic [ 3.947934] usbserial: USB Serial support registered for generic [ 3.950447] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.954685] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.956965] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.959636] mousedev: PS/2 mouse device common for all mice [ 3.963125] rtc_cmos 00:05: RTC can wake from S4 [ 3.965687] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.970275] rtc_cmos 00:05: registered as rtc0 [ 3.971929] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.973182] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.979329] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.980187] intel_pstate: CPU model not supported [ 3.987143] hid: raw HID events driver (C) Jiri Kosina [ 3.989493] usbcore: registered new interface driver usbhid [ 3.991380] usbhid: USB HID core driver [ 3.993094] drop_monitor: Initializing network drop monitor service [ 3.995579] Initializing XFRM netlink socket [ 3.997425] NET: Registered protocol family 10 [ 4.000294] Segment Routing with IPv6 [ 4.001890] NET: Registered protocol family 17 [ 4.004268] mpls_gso: MPLS GSO support [ 4.009159] RAS: Correctable Errors collector initialized. [ 4.011590] AVX version of gcm_enc/dec engaged. [ 4.012698] AES CTR mode by8 optimization enabled [ 4.087255] sched_clock: Marking stable (4087227879, 0)->(4968324782, -881096903) [ 4.090473] registered taskstats version 1 [ 4.092359] Loading compiled-in X.509 certificates [ 4.096402] zswap: loaded using pool lzo/zbud [ 4.121831] Key type big_key registered [ 4.134949] Key type encrypted registered [ 4.136677] ima: No TPM chip found, activating TPM-bypass! [ 4.138657] ima: Allocated hash algorithm: sha1 [ 4.140484] ima: No architecture policies found [ 4.144377] evm: Initialising EVM extended attributes: [ 4.146234] evm: security.selinux [ 4.147205] evm: security.ima [ 4.148272] evm: security.capability [ 4.149580] evm: HMAC attrs: 0x1 [ 4.151617] rtc_cmos 00:05: setting system clock to 2025-11-17 00:43:03 UTC (1763340183) [ 4.157287] debug: unmapping init [mem 0xffffffffad203000-0xffffffffad3fffff] [ 4.160770] debug: unmapping init [mem 0xffffffffabf82000-0xffffffffac258fff] [ 4.177076] Write protecting the kernel read-only data: 28672k [ 4.180231] debug: unmapping init [mem 0xffffffffaa603000-0xffffffffaa7fffff] [ 4.182939] debug: unmapping init [mem 0xffffffffaaf14000-0xffffffffaaffffff] [ 4.220668] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.229380] systemd[1]: Detected virtualization kvm. [ 4.231469] systemd[1]: Detected architecture x86-64. [ 4.233375] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.260324] systemd[1]: No hostname configured. [ 4.262502] systemd[1]: Set hostname to . [ 4.264495] random: systemd: uninitialized urandom read (16 bytes read) [ 4.266505] systemd[1]: Initializing machine ID from random generator. [ 4.325826] random: ln: uninitialized urandom read (6 bytes read) [ 4.417312] random: systemd: uninitialized urandom read (16 bytes read) [ 4.419773] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.423755] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.428131] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.098808] device-mapper: uevent: version 1.0.3 [ 5.100678] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.857333] random: fast init done [ 5.897102] virtio_net virtio0 ens2: renamed from eth0 [ 5.943328] scsi host0: ata_piix [ 6.030161] scsi host1: ata_piix [ 6.031752] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.034279] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.725210] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.654672] random: crng init done [ 10.655969] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.218074] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.443487] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.689553] SELinux: Disabled at runtime. [ 12.754614] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.761820] systemd[1]: Detected virtualization kvm. [ 12.763309] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.235411] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.239290] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.244800] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.249479] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.252725] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.262696] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.274450] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 13.391690] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.797183] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.112379] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.157248] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.301446] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.312687] EDAC sbridge: Ver: 1.1.2 [ 16.037451] Key type dns_resolver registered [ 16.339950] NFS: Registering the id_resolver key type [ 16.341968] Key type id_resolver registered [ 16.343561] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg159-server login: [ 52.402476] libcfs: loading out-of-tree module taints kernel. [ 52.440310] Key type ._llcrypt registered [ 52.441785] Key type .llcrypt registered [ 52.553400] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 68.575398] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 69.627212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 69.636489] alg: No test for adler32 (adler32-zlib) [ 70.854638] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 71.350481] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 73.040568] Key type lgssc registered [ 74.425357] Lustre: Echo OBD driver; http://www.lustre.org/ [ 85.001855] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 86.375343] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 92.864080] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 98.014765] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 103.363752] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 113.577234] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 123.712385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 123.765837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 123.778995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 125.062256] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 125.107924] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 125.193646] Lustre: lustre-MDT0000: new disk, initializing [ 125.274297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 125.290185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 128.880255] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 140.059915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 140.121020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 140.165025] Lustre: 6511:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 140.198912] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 140.201523] Lustre: Skipped 1 previous similar message [ 140.260052] Lustre: lustre-MDT0001: new disk, initializing [ 140.331166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 140.356859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 140.362798] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 143.635710] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 148.026785] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 156.579320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 156.635386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 156.819685] Lustre: lustre-OST0000: new disk, initializing [ 156.829061] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 156.886498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 158.296366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 158.302383] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 158.355738] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 161.184036] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 171.615665] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 171.673792] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 171.738214] Lustre: lustre-OST0001: new disk, initializing [ 171.740710] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 171.781415] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 176.171009] hrtimer: interrupt took 4009810 ns [ 176.253925] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 181.258606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 181.265169] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 181.302060] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 184.721904] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 188.914168] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 197.515839] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing check_logdir /tmp/testlogs/ [ 200.206596] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing yml_node [ 203.631366] Lustre: DEBUG MARKER: Client: 2.16.61.49 [ 205.797585] Lustre: DEBUG MARKER: MDS: 2.16.61.49 [ 207.760399] Lustre: DEBUG MARKER: OSS: 2.16.61.49 [ 208.784247] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Nov 16 19:46:26 EST 2025 [ 219.967695] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 221.052651] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 237.538253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 237.538793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 237.539217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 237.548138] Lustre: Skipped 3 previous similar messages [ 240.659914] LustreError: 12216:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 242.723601] LustreError: 12216:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 242.728191] LustreError: 12216:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 242.762489] Lustre: server umount lustre-MDT0000 complete [ 245.249382] LustreError: 8419:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340424 with bad export cookie 16303252562165085409 [ 245.251871] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 245.253819] LustreError: 8419:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 247.777567] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 247.782547] LustreError: 6516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 247.786321] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 247.794016] Lustre: Skipped 3 previous similar messages [ 247.797026] LustreError: 6516:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 250.786401] LustreError: 6516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 250.787235] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 250.802649] Lustre: Skipped 2 previous similar messages [ 251.446147] LustreError: 12417:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 251.450106] LustreError: 12417:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 251.646886] Lustre: server umount lustre-MDT0001 complete [ 254.283136] LustreError: 12618:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 254.293895] LustreError: 12618:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 254.337376] Lustre: server umount lustre-OST0000 complete [ 257.583861] Lustre: server umount lustre-OST0001 complete [ 262.377837] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 270.994381] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 280.027334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 284.938909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 288.986974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 293.213070] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 298.152899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 298.199836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 298.358352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 298.384961] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 298.430864] Lustre: lustre-MDT0000: new disk, initializing [ 298.519856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 298.535758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 301.087788] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 308.073049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 308.114182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 308.174516] Lustre: 15001:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 308.206359] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 308.208098] Lustre: Skipped 1 previous similar message [ 308.252232] Lustre: lustre-MDT0001: new disk, initializing [ 308.313499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 308.336027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 308.343819] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 310.474101] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 315.358971] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 317.361541] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 321.360972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 321.398790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 322.976451] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 323.057170] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 325.115567] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 329.814778] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 332.121352] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 332.243859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 334.753234] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 334.928552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 338.404574] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 338.405033] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.409597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 338.412501] LustreError: Skipped 1 previous similar message [ 338.416637] Lustre: Skipped 2 previous similar messages [ 342.320605] LustreError: 17941:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 342.325427] LustreError: 17941:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 342.380495] Lustre: server umount lustre-OST0000 complete [ 348.643110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 348.648867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 348.654835] Lustre: Skipped 1 previous similar message [ 348.664603] Lustre: Skipped 1 previous similar message [ 357.857836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 357.861782] Lustre: Skipped 3 previous similar messages [ 359.392149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 359.461420] LustreError: 18142:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 359.464261] LustreError: 18142:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 359.519758] Lustre: server umount lustre-MDT0000 complete [ 362.068922] LustreError: 14994:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340541 with bad export cookie 16303252562165086942 [ 362.073837] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 362.076781] LustreError: 14994:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 362.292362] Lustre: server umount lustre-MDT0001 complete [ 366.477269] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 19:49:04 (1763340544) [ 371.397089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 371.730109] LustreError: 18872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 371.740835] LustreError: 18872:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 371.779656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 371.783775] Lustre: Skipped 1 previous similar message [ 374.413599] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 376.801235] LustreError: 18872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 379.898829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 383.071663] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 387.054519] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 389.563129] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 394.432662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 398.485609] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 402.018542] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 408.329288] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 414.844241] LustreError: 21171:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 414.848455] LustreError: 21171:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 414.918979] Lustre: server umount lustre-MDT0000 complete [ 415.205073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 415.212072] LustreError: 20652:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 415.213013] Lustre: Skipped 1 previous similar message [ 415.227909] LustreError: 20652:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 417.687665] LustreError: 18853:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340597 with bad export cookie 16303252562165087824 [ 417.690975] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 417.692718] LustreError: 18853:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 417.899941] Lustre: server umount lustre-MDT0001 complete [ 430.490776] Lustre: server umount lustre-OST0000 complete [ 443.079512] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 450.240859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 450.564684] LustreError: 22898:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 450.635105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 450.638018] Lustre: Skipped 2 previous similar messages [ 453.353134] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 459.595652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 459.747167] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 463.249157] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 465.536846] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 470.724938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 475.344411] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 481.662685] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 481.714273] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 481.879870] Lustre: lustre-OST0001: new disk, initializing [ 481.882806] Lustre: Skipped 1 previous similar message [ 481.886725] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 481.891038] Lustre: Skipped 1 previous similar message [ 483.330572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 483.336712] Lustre: Skipped 1 previous similar message [ 483.341262] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 483.423940] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 486.325881] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 493.752141] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 496.214592] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 509.296816] LustreError: 26378:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 509.298858] LustreError: 26378:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 509.362361] Lustre: server umount lustre-MDT0000 complete [ 510.944895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 510.959065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.973464] Lustre: Skipped 4 previous similar messages [ 510.980484] LustreError: 22893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 510.997744] LustreError: 22893:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 512.226398] LustreError: 25167:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340691 with bad export cookie 16303252562165088671 [ 512.227231] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 512.235681] LustreError: 25167:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 514.545683] Lustre: server umount lustre-MDT0001 complete [ 528.169495] Lustre: server umount lustre-OST0000 complete [ 541.453043] Lustre: server umount lustre-OST0001 complete [ 553.809421] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 561.010435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 561.384314] LustreError: 28153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 561.441590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 561.448471] Lustre: Skipped 3 previous similar messages [ 564.464375] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 572.181909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 575.330822] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 577.622122] Lustre: 29269:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 582.408640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 587.079140] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 593.671700] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 599.192637] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 606.493179] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 609.341691] Lustre: 31125:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 623.080538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 623.088739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 623.094677] Lustre: Skipped 2 previous similar messages [ 623.131635] Lustre: Skipped 3 previous similar messages [ 625.002265] LustreError: 31327:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 625.005919] LustreError: 31327:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 625.103796] Lustre: server umount lustre-MDT0000 complete [ 627.749268] LustreError: 28134:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340807 with bad export cookie 16303252562165089966 [ 627.753953] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 627.759432] LustreError: 28134:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 627.996374] Lustre: server umount lustre-MDT0001 complete [ 641.452975] Lustre: server umount lustre-OST0000 complete [ 644.449141] Lustre: 3656:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763340807/real 1763340807] req@ffff9d026d604380 x1848996266242432/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763340823 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 644.655292] Lustre: server umount lustre-OST0001 complete [ 651.450936] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 19:53:48 (1763340828) [ 674.876682] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 684.547463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 685.003187] LustreError: 33912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 685.025438] LustreError: 33912:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 689.391976] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 697.071432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 697.489808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 697.501092] Lustre: Skipped 4 previous similar messages [ 701.127653] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 703.308567] Lustre: 35028:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 707.733159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 711.946532] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 718.339378] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 722.655563] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 729.049120] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 732.234340] Lustre: 36881:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 742.886319] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 742.891534] Lustre: Skipped 2 previous similar messages [ 742.894802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 742.897568] Lustre: Skipped 2 previous similar messages [ 748.124311] Lustre: server umount lustre-MDT0000 complete [ 750.970947] LustreError: 33892:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340930 with bad export cookie 16303252562165091072 [ 750.974649] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 750.983692] LustreError: 33892:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 751.228665] Lustre: server umount lustre-MDT0001 complete [ 764.244093] LustreError: 37438:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 764.258194] LustreError: 37438:0:(obd_class.h:479:obd_check_dev()) Skipped 41 previous similar messages [ 764.322949] Lustre: server umount lustre-OST0000 complete [ 777.519997] Lustre: server umount lustre-OST0001 complete [ 784.117769] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 19:56:01 (1763340961) [ 794.572495] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 802.720874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 808.291894] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 813.540883] LustreError: 39007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 813.580164] LustreError: 39007:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 817.715483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 820.909565] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 823.246874] Lustre: 40125:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 828.818920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 834.022125] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 839.873024] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 844.908852] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 857.347324] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 861.222934] Lustre: 41981:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 867.079379] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 867.082525] Lustre: Skipped 1 previous similar message [ 873.130026] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.59@tcp (stopping) [ 873.138346] Lustre: Skipped 4 previous similar messages [ 873.956948] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 873.980210] Lustre: Skipped 6 previous similar messages [ 875.921904] Lustre: server umount lustre-MDT0000 complete [ 879.245764] LustreError: 38987:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341058 with bad export cookie 16303252562165092178 [ 879.249576] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 879.261579] LustreError: 38987:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 885.616787] Lustre: server umount lustre-MDT0001 complete [ 891.880115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 904.676454] LustreError: 3652:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9d01435cd500 x1848996266342656/t0(0) o250->MGC192.168.201.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 908.546425] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 914.360577] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 915.786680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 920.101465] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 925.684252] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 925.687459] Lustre: Skipped 1 previous similar message [ 926.327561] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 927.843486] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 930.812500] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 932.216447] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 936.420135] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 942.492358] Lustre: server umount lustre-MDT0000 complete [ 945.730845] LustreError: 38989:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341125 with bad export cookie 16303252562165093368 [ 945.743069] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 945.746338] LustreError: 38989:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 946.124545] Lustre: server umount lustre-MDT0001 complete [ 959.725124] Lustre: server umount lustre-OST0000 complete [ 962.656211] Lustre: 3655:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763341125/real 1763341125] req@ffff9d0274c81f80 x1848996266373248/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763341141 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 963.797958] Lustre: server umount lustre-OST0001 complete [ 973.024853] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 19:59:10 (1763341150) [ 987.281757] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 997.278342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 997.769526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 997.772264] Lustre: Skipped 8 previous similar messages [ 1001.371964] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1008.948265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1013.122249] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1015.884891] Lustre: 47480:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1021.853351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1027.186456] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1034.213680] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1039.241856] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1045.245213] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1054.970941] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1065.444065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1065.448927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1065.483865] Lustre: Skipped 10 previous similar messages [ 1067.379786] LustreError: 49560:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 1067.393896] LustreError: 49560:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 1067.547818] Lustre: server umount lustre-MDT0000 complete [ 1070.562732] LustreError: 46360:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1070.574762] LustreError: 46360:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 1071.031896] LustreError: 49345:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341250 with bad export cookie 16303252562165094152 [ 1071.042862] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1071.058326] LustreError: 49345:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1071.318440] Lustre: server umount lustre-MDT0001 complete [ 1085.039206] Lustre: server umount lustre-OST0000 complete [ 1099.220572] Lustre: server umount lustre-OST0001 complete [ 1108.180699] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 20:01:25 (1763341285) [ 1110.159207] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1111.951242] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 20:01:29 (1763341289) [ 1123.371614] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 1131.384939] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 1140.410748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1146.065210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1152.222527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1152.291190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1152.535978] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1152.591557] Lustre: lustre-MDT0000: new disk, initializing [ 1152.672606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1155.730513] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1162.728271] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1169.176720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1169.234230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1169.419549] Lustre: lustre-OST0000: new disk, initializing [ 1169.421967] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1169.424736] Lustre: Skipped 1 previous similar message [ 1171.448426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1171.456306] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1171.493464] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1174.718826] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1182.872781] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1185.330960] Lustre: DEBUG MARKER: create test files [ 1217.505268] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1217.521321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1217.526445] Lustre: Skipped 16 previous similar messages [ 1221.660162] Lustre: server umount lustre-OST0000 complete [ 1224.938100] Lustre: server umount lustre-MDT0000 complete [ 1233.822078] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 1236.238242] Key type lgssc unregistered [ 1236.443482] LNet: 54912:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1236.448805] LNetError: 54912:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1236.472127] LNet: Removed LNI 192.168.201.159@tcp [ 1237.100469] Key type .llcrypt unregistered [ 1237.103135] Key type ._llcrypt unregistered [ 1263.213331] Key type ._llcrypt registered [ 1263.214527] Key type .llcrypt registered [ 1263.331757] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 1264.847737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1264.911249] alg: No test for adler32 (adler32-zlib) [ 1265.985830] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 1266.258140] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 1268.008411] Key type lgssc registered [ 1269.819420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1280.537246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1280.580285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1282.209598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1286.602866] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1292.638548] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1298.488805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1298.799685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1303.823668] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1306.998858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:100 to 0x240000400:129) [ 1307.089731] Lustre: 57518:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1307.094534] Lustre: 57518:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1307.098432] Lustre: 57518:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1307.102177] Lustre: 57518:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 1307.106887] Lustre: 57518:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1307.111113] Lustre: 57518:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1307.807928] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1313.713784] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1318.449630] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1353.186503] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1353.196966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1353.213186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1355.897278] LustreError: 58291:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1355.995045] Lustre: server umount lustre-OST0000 complete [ 1359.424790] LustreError: 58492:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1359.427149] LustreError: 58492:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1359.615560] Lustre: server umount lustre-MDT0000 complete [ 1369.582925] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 1371.872686] Key type lgssc unregistered [ 1372.173309] LNet: 59026:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1372.182470] LNetError: 59026:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1372.197216] LNet: Removed LNI 192.168.201.159@tcp [ 1372.994428] Key type .llcrypt unregistered [ 1372.999795] Key type ._llcrypt unregistered [ 1401.672456] Key type ._llcrypt registered [ 1401.674124] Key type .llcrypt registered [ 1401.755288] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 1403.183528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1403.204696] alg: No test for adler32 (adler32-zlib) [ 1404.269848] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 1404.441137] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 1406.096206] Key type lgssc registered [ 1407.205729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1416.153731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1416.173603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1417.646487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1422.308318] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1428.234142] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1434.412779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1434.662932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1440.030124] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1441.847419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1441.966896] Lustre: 61725:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1441.979941] Lustre: 61725:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1441.991751] Lustre: 61725:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1441.995845] Lustre: 61725:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 1442.000660] Lustre: 61725:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1442.009966] Lustre: 61725:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1444.948103] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1450.116061] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1487.842979] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1487.847123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1487.861705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1490.921104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1493.459982] LustreError: 62443:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1493.544106] Lustre: server umount lustre-OST0000 complete [ 1497.102790] LustreError: 62645:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1497.111882] LustreError: 62645:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1497.312832] Lustre: server umount lustre-MDT0000 complete [ 1507.606986] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 1510.030725] Key type lgssc unregistered [ 1510.293478] LNet: 63178:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1510.302605] LNetError: 63178:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1510.339594] LNet: Removed LNI 192.168.201.159@tcp [ 1511.155212] Key type .llcrypt unregistered [ 1511.157708] Key type ._llcrypt unregistered [ 1527.881732] Key type ._llcrypt registered [ 1527.883976] Key type .llcrypt registered [ 1527.996814] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 1541.206814] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 1542.291261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1542.374485] alg: No test for adler32 (adler32-zlib) [ 1543.364605] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 1543.544992] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 1545.208162] Key type lgssc registered [ 1546.170672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1554.254182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1559.568863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1564.270600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1569.224994] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1574.132615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1574.187450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1574.211320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1575.403897] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1575.440915] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1575.510613] Lustre: lustre-MDT0000: new disk, initializing [ 1575.581442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1575.591763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1578.448886] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1587.742729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1587.850345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1587.957875] Lustre: 66305:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1588.005961] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1588.009513] Lustre: Skipped 1 previous similar message [ 1588.114931] Lustre: lustre-MDT0001: new disk, initializing [ 1588.246905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1588.303320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1588.311989] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1591.928659] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1598.638471] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1602.230072] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1608.289600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1608.346887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1608.545474] Lustre: lustre-OST0000: new disk, initializing [ 1608.548254] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1608.630980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1613.138502] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1614.400336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1614.406629] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1614.488200] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1618.508206] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1620.917066] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1621.060372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1623.721486] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1623.836354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1625.058907] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1625.061278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1625.063759] LustreError: Skipped 1 previous similar message [ 1625.078038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1625.082338] Lustre: Skipped 1 previous similar message [ 1630.180745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1630.190308] Lustre: Skipped 1 previous similar message [ 1635.300815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1635.306501] Lustre: Skipped 1 previous similar message [ 1639.392374] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1639.439078] LustreError: 69243:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1639.580459] Lustre: server umount lustre-OST0000 complete [ 1644.517497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1644.521205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1644.530894] Lustre: Skipped 1 previous similar message [ 1644.536896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1645.541760] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1648.713678] LustreError: 69447:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1648.719728] LustreError: 69447:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1648.811682] Lustre: server umount lustre-MDT0000 complete [ 1651.692977] LustreError: 66296:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341831 with bad export cookie 6275307268911524348 [ 1651.700096] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1651.707907] LustreError: 66296:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1651.834513] LustreError: 69648:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1651.838627] LustreError: 69648:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1651.951470] Lustre: server umount lustre-MDT0001 complete [ 1658.956846] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 20:10:36 (1763341836) [ 1667.845611] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 1678.158694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1699.769464] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 1707.650989] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 1716.376667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1722.449916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1727.866641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1733.928872] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1739.478278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1739.537377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1739.711099] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1739.748799] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1739.816909] Lustre: lustre-MDT0000: new disk, initializing [ 1739.869619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1739.886724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1743.139315] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1752.742543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1752.790399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1752.885100] Lustre: 73886:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1752.941081] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1752.949243] Lustre: Skipped 1 previous similar message [ 1753.016651] Lustre: lustre-MDT0001: new disk, initializing [ 1753.106293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1753.146656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1753.155044] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1755.966211] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1762.122680] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1764.847447] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1769.568619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1769.624484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1769.812337] Lustre: lustre-OST0000: new disk, initializing [ 1769.819706] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1769.881265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1774.494777] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1778.739744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1778.754594] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1778.881387] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1780.282757] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1783.773744] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1783.923511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1786.864686] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1787.041375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1789.409889] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1789.412080] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1789.420501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1789.423119] Lustre: Skipped 1 previous similar message [ 1789.432409] Lustre: Skipped 3 previous similar messages [ 1794.561119] LustreError: 76822:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1794.566839] LustreError: 76822:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1794.671934] Lustre: server umount lustre-OST0000 complete [ 1799.139256] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1799.145445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1799.159176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1799.163119] Lustre: Skipped 2 previous similar messages [ 1803.538682] LustreError: 77024:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1803.543785] LustreError: 77024:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1803.613374] Lustre: server umount lustre-MDT0000 complete [ 1804.775092] LustreError: 74680:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1804.781309] LustreError: 74680:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1806.756145] LustreError: 75631:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341986 with bad export cookie 6275307268911525230 [ 1806.757536] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1806.769152] LustreError: 75631:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1807.089536] Lustre: server umount lustre-MDT0001 complete [ 1813.720887] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 20:13:11 (1763341991) [ 1818.683965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1818.973523] LustreError: 77897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1819.036428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1822.238418] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1824.233101] LustreError: 77896:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1824.243092] LustreError: 77896:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1828.667217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1832.117546] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1836.897441] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1839.855539] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1845.824733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1850.809109] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1855.115766] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1857.749290] Lustre: 80234:0:(genops.c:1791:obd_export_evict_by_uuid()) MGS: evicting f4d71354-dfa7-4c51-8e74-cef9ff341ad6 at adminstrative request [ 1859.557132] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1859.567445] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 1868.772264] Lustre: MGS: Client f4d71354-dfa7-4c51-8e74-cef9ff341ad6 (at 0@lo) reconnecting [ 1868.778271] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x57165eb68be5f0d9 to 0x57165eb68be5f379 [ 1868.786215] Lustre: MGC192.168.201.159@tcp: Connection restored to 0@lo (at 0@lo) [ 1893.215966] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1893.257303] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1893.356989] Lustre: lustre-OST0001: new disk, initializing [ 1893.363671] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1893.436891] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1893.441696] Lustre: Skipped 2 previous similar messages [ 1897.972375] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1903.164192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1903.171829] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1903.245255] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1903.554614] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1908.201954] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1908.204171] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1908.207630] Lustre: Skipped 2 previous similar messages [ 1908.214052] Lustre: Skipped 1 previous similar message [ 1911.947131] LustreError: 81372:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 1911.953180] LustreError: 81372:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1912.002466] Lustre: server umount lustre-OST0001 complete [ 1913.316493] LustreError: 81196:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1913.335941] LustreError: 81196:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1915.873581] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1918.436758] LustreError: 79494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1918.457477] LustreError: 79494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1921.472067] Lustre: server umount lustre-OST0000 complete [ 1926.115939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1926.121067] LustreError: Skipped 1 previous similar message [ 1926.123878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1926.134933] Lustre: Skipped 2 previous similar messages [ 1929.970754] LustreError: 81777:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1929.972941] LustreError: 81777:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1930.025495] Lustre: server umount lustre-MDT0000 complete [ 1932.754802] LustreError: 77878:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342112 with bad export cookie 6275307268911526777 [ 1932.756752] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1932.759669] LustreError: 77878:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1932.983305] Lustre: server umount lustre-MDT0001 complete [ 1939.436848] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 20:15:17 (1763342117) [ 1940.837580] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 1942.229743] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 20:15:19 (1763342119) [ 1943.746269] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 1945.138856] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 20:15:22 (1763342122) [ 1946.397675] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 1947.998730] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 20:15:25 (1763342125) [ 1949.287221] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 1953.659340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1954.572584] loop: module loaded [ 1954.586519] loop0: detected capacity change from 0 to 4294967296 [ 1954.658064] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 1956.113963] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 1958.946643] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 1962.792468] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 1970.296701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1974.051907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1977.305109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1980.685112] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1986.026498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1986.081609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1986.102865] LustreError: 84555:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-28) [ 1986.106510] LustreError: 84555:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 1986.110053] LustreError: 84555:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 1986.118043] LustreError: 84555:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2001.729573] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 2009.951532] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 2018.828417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2025.017719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2030.500762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2035.847486] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2041.762857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2041.820742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2042.029773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2042.060813] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2042.125420] Lustre: lustre-MDT0000: new disk, initializing [ 2042.209131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2042.222548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2045.691898] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2055.028465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2055.097671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2055.188767] Lustre: 87398:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2055.517943] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2059.018050] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2065.248746] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2067.835725] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2073.618939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2073.678958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2079.953745] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2080.828968] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2080.959129] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2087.681633] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2091.230134] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2091.435884] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2095.158736] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2095.320472] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2101.729927] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2101.734905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2101.745533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2101.761450] Lustre: Skipped 2 previous similar messages [ 2101.764149] Lustre: Skipped 8 previous similar messages [ 2103.108308] LustreError: 90335:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2103.111393] LustreError: 90335:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2103.219646] Lustre: server umount lustre-OST0000 complete [ 2112.982711] Lustre: server umount lustre-MDT0000 complete [ 2116.275064] LustreError: 89141:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342295 with bad export cookie 6275307268911527379 [ 2116.279935] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2116.289477] LustreError: 89141:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2116.571057] Lustre: server umount lustre-MDT0001 complete [ 2123.890656] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 20:18:21 (1763342301) [ 2129.814831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.190447] LustreError: 91411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2134.717921] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2144.329633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2149.583916] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2154.379344] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2157.663543] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2163.297580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2168.518520] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2173.087692] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2176.644330] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2176.818473] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2180.334299] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2180.553982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2186.042305] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2186.148527] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2186.272583] Lustre: lustre-OST0001: new disk, initializing [ 2186.274753] Lustre: Skipped 2 previous similar messages [ 2186.285885] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2186.295201] Lustre: Skipped 3 previous similar messages [ 2186.403268] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2186.409804] Lustre: Skipped 5 previous similar messages [ 2191.191323] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2194.515160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2194.520368] Lustre: Skipped 2 previous similar messages [ 2194.534036] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2194.613643] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2198.076420] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2201.899086] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2202.132281] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2206.016978] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2206.240348] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2222.544393] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2249.452708] Lustre: Failing over lustre-MDT0000 [ 2251.241093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2251.242565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2251.256454] Lustre: Skipped 5 previous similar messages [ 2251.262786] Lustre: Skipped 8 previous similar messages [ 2251.893221] LustreError: 96325:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2251.903703] LustreError: 96325:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2252.022292] Lustre: server umount lustre-MDT0000 complete [ 2254.500780] LustreError: 91407:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2254.511038] LustreError: 91407:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2272.736153] Lustre: 64516:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763342435/real 1763342435] req@000000005eca121c x1848997809732736/t0(0) o400->MGC192.168.201.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763342451 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2272.757478] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2274.934296] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2274.937173] LDISKFS-fs (dm-0): recovery complete [ 2274.948842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2282.984776] LustreError: 64514:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bc81aec9 x1848997809741184/t0(0) o250->MGC192.168.201.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2283.316092] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2286.232137] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2286.245239] Lustre: 91406:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2286.830098] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2288.613653] Lustre: 91411:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2288.627290] Lustre: 91411:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 1 previous similar message [ 2288.650564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2288.737801] LustreError: 97075:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2289.056588] LustreError: 97075:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2289.390885] LustreError: 97075:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2289.404961] LustreError: 97075:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 1 previous similar message [ 2289.728156] LustreError: 97075:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2289.738501] LustreError: 97075:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 1 previous similar message [ 2289.758801] Lustre: 97075:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2289.776367] Lustre: 97075:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 7 previous similar messages [ 2290.433263] LustreError: 97075:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2290.440039] LustreError: 97075:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 2 previous similar messages [ 2290.769054] LustreError: 97075:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2290.772020] LustreError: 97075:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 2 previous similar messages [ 2291.765450] Lustre: 97075:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2291.780413] Lustre: 97075:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 5 previous similar messages [ 2292.747122] LustreError: 97075:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2292.759117] LustreError: 97075:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 6 previous similar messages [ 2293.080186] LustreError: 97075:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2293.086355] LustreError: 97075:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 6 previous similar messages [ 2295.980228] Lustre: 97075:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2295.990975] Lustre: 97075:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 12 previous similar messages [ 2296.951354] LustreError: 97075:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2296.968410] LustreError: 97075:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 12 previous similar messages [ 2297.296169] LustreError: 97075:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2297.304111] LustreError: 97075:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 12 previous similar messages [ 2304.020979] Lustre: 97075:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2304.031678] Lustre: 97075:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 24 previous similar messages [ 2305.022525] LustreError: 97075:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2305.032861] LustreError: 97075:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 24 previous similar messages [ 2305.356480] LustreError: 97075:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2305.364808] LustreError: 97075:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 24 previous similar messages [ 2320.263882] Lustre: 97075:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2320.273233] Lustre: 97075:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 50 previous similar messages [ 2321.232963] LustreError: 97075:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2321.239877] LustreError: 97075:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 50 previous similar messages [ 2321.560119] LustreError: 97075:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2321.564378] LustreError: 97075:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 50 previous similar messages [ 2346.264173] Lustre: 97075:0:(ldlm_lib.c:2375:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2346.272608] Lustre: 97075:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2346.287561] Lustre: 97075:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client ffacc453-76d0-4ae7-8330-bd46d1fd0fce@192.168.201.59@tcp [ 2346.301907] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2346.309777] Lustre: 97075:0:(ldlm_lib.c:1897:abort_req_replay_queue()) @@@ aborted: req@0000000022d0f866 x1848997795926400/t0(8589934776) o35->ffacc453-76d0-4ae7-8330-bd46d1fd0fce@192.168.201.59@tcp:21/0 lens 392/0 e 0 to 0 dl 1763342536 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2346.348120] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2346.363822] Lustre: lustre-MDT0000: Denying connection for new client ffacc453-76d0-4ae7-8330-bd46d1fd0fce (at 192.168.201.59@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) already passed deadline 0:00 [ 2346.387381] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2346.418980] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2346.423431] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2346.424825] Lustre: Skipped 2 previous similar messages [ 2346.460913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:79 to 0x280000401:577) [ 2346.465753] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:47 to 0x2c0000401:545) [ 2357.219051] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2357.223813] LustreError: Skipped 2 previous similar messages [ 2361.681532] Lustre: server umount lustre-OST0000 complete [ 2365.414984] LustreError: 97436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2365.429453] LustreError: 97436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 41 previous similar messages [ 2371.440763] Lustre: server umount lustre-OST0001 complete [ 2380.647567] LustreError: 98137:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 2380.657266] LustreError: 98137:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 2380.767807] Lustre: server umount lustre-MDT0000 complete [ 2383.836398] LustreError: 91390:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342563 with bad export cookie 6275307268911578598 [ 2383.852643] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2384.109543] Lustre: server umount lustre-MDT0001 complete [ 2391.045345] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 20:22:48 (1763342568) [ 2397.467612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2397.471674] Lustre: *** cfs_fail_loc=197, val=0*** [ 2413.024691] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2418.144603] LustreError: 99031:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.159@tcp: failed processing log, type 4: rc = -110 [ 2423.302039] Lustre: *** cfs_fail_loc=197, val=0*** [ 2423.305550] Lustre: Skipped 65 previous similar messages [ 2443.873583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2443.881981] Lustre: Skipped 1 previous similar message [ 2448.641398] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2452.982622] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2454.567522] Lustre: server umount lustre-OST0000 complete [ 2461.420071] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 20:23:59 (1763342639) [ 2480.682322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2480.999344] LustreError: 100538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2481.018661] LustreError: 100538:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2484.228611] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2490.825532] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2494.374550] LustreError: 100568:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2494.390108] LustreError: 100568:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2494.395945] LustreError: 100568:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 13, retries 0, failed: rc = -5 [ 2494.637814] Lustre: server umount lustre-MDT0000 complete [ 2507.428973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2519.852709] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 2526.867332] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 2533.577074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2538.333428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2543.592392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2549.383623] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2554.718548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2554.778265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2554.982776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2555.012298] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2555.084733] Lustre: lustre-MDT0000: new disk, initializing [ 2555.170218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2558.162602] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2566.823951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2566.906539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2566.993631] Lustre: 104608:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2567.221049] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2570.449398] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2577.561210] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2580.661899] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2586.212306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2586.279803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2591.018160] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2593.902259] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2596.963081] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2600.494899] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2600.713529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2603.908787] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2604.131874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2609.126057] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2609.130725] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2609.130735] Lustre: Skipped 6 previous similar messages [ 2609.131519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2609.131523] Lustre: Skipped 10 previous similar messages [ 2609.145702] LustreError: Skipped 2 previous similar messages [ 2611.904863] Lustre: server umount lustre-OST0000 complete [ 2621.982293] Lustre: server umount lustre-MDT0000 complete [ 2624.486833] LustreError: 104615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2624.506772] LustreError: 104615:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2625.722980] LustreError: 104600:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342805 with bad export cookie 6275307268911589231 [ 2625.728985] LustreError: 104600:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2625.733957] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2626.076321] Lustre: server umount lustre-MDT0001 complete [ 2634.160845] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 20:26:51 (1763342811) [ 2640.745414] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 2642.961794] Key type lgssc unregistered [ 2643.186631] LNet: 108680:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2643.201225] LNetError: 108680:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2643.213795] LNet: Removed LNI 192.168.201.159@tcp [ 2643.978250] Key type .llcrypt unregistered [ 2643.984352] Key type ._llcrypt unregistered [ 2659.756394] Key type ._llcrypt registered [ 2659.758235] Key type .llcrypt registered [ 2659.845638] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 2673.129475] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 2674.225671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2674.262142] alg: No test for adler32 (adler32-zlib) [ 2675.329648] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 2675.598356] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 2677.264184] Key type lgssc registered [ 2678.305353] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2685.703210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2691.047257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2697.239602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2704.096677] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2710.170254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2717.464035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2717.563074] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2717.594845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2719.029440] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2719.083293] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2719.173667] Lustre: lustre-MDT0000: new disk, initializing [ 2719.286173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2719.309624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2723.592971] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2732.930661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2732.984784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2733.168794] Lustre: lustre-OST0000: new disk, initializing [ 2733.174240] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2733.182950] Lustre: Skipped 1 previous similar message [ 2733.249339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2733.714039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2733.726911] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2733.776177] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2737.609931] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2746.346872] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2746.400061] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2746.487557] Lustre: lustre-OST0001: new disk, initializing [ 2746.496809] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2746.567209] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2751.611267] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2756.681473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 2756.696057] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 2756.744084] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 2761.700364] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2761.721352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2761.727718] Lustre: Skipped 1 previous similar message [ 2765.837671] LustreError: 114257:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 2766.029549] Lustre: server umount lustre-MDT0000 complete [ 2772.518913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2777.477507] LustreError: 112016:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342956 with bad export cookie 17363800710716967519 [ 2777.479348] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2777.503502] LustreError: 112016:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2777.587891] LustreError: 114852:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 2777.590890] LustreError: 114852:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2777.624313] Lustre: server umount lustre-OST0000 complete [ 2781.938212] LustreError: 115053:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2781.940724] LustreError: 115053:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2782.113067] Lustre: server umount lustre-OST0001 complete [ 2792.720495] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 20:29:29 (1763342969) [ 2803.051901] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 2811.013708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2812.055911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2817.005994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2818.100742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2824.107161] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 20:30:01 (1763343001) [ 2835.238542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2836.953110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2838.880801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2840.681668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2841.611907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2842.980199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2856.356291] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 2863.746125] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 2870.347129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2875.182612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2880.979857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2886.855490] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2893.459161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2893.510177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2893.687824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2893.711756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2893.773388] Lustre: lustre-MDT0000: new disk, initializing [ 2893.858731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2893.876235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2897.066214] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2905.663682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2905.719523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2905.785698] Lustre: 120379:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2905.834937] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2905.838428] Lustre: Skipped 1 previous similar message [ 2905.884946] Lustre: lustre-MDT0001: new disk, initializing [ 2905.971169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2906.001839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2906.013789] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2908.676611] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2914.219340] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2917.290173] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2922.512654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2922.583077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2922.749750] Lustre: lustre-OST0000: new disk, initializing [ 2922.752370] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2922.828104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2927.435046] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2929.202332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2929.212700] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2929.301096] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2933.472726] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2936.386373] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2936.510848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2939.217708] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2939.353201] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2945.003383] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2945.003755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2945.008595] Lustre: Skipped 1 previous similar message [ 2945.009578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2945.015737] LustreError: Skipped 1 previous similar message [ 2945.017973] Lustre: Skipped 1 previous similar message [ 2946.741243] LustreError: 123323:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2946.765057] LustreError: 123323:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2946.835336] Lustre: server umount lustre-OST0000 complete [ 2950.119115] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2950.129891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2950.131189] Lustre: Skipped 2 previous similar messages [ 2950.150411] Lustre: Skipped 1 previous similar message [ 2955.234159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2955.238727] Lustre: Skipped 1 previous similar message [ 2955.869877] LustreError: 123525:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2955.876716] LustreError: 123525:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2955.995102] Lustre: server umount lustre-MDT0000 complete [ 2958.932728] LustreError: 120372:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343138 with bad export cookie 17363800710716968737 [ 2958.936587] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2958.942932] LustreError: 120372:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2959.244638] Lustre: server umount lustre-MDT0001 complete [ 2967.112049] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 20:32:24 (1763343144) [ 2972.274802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2972.650262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2975.408964] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2977.761715] LustreError: 124398:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2977.771603] LustreError: 124398:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2981.874571] LustreError: 124398:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2983.088488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2987.439607] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2993.630173] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2997.502415] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3003.298032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3008.475476] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3013.314526] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3022.968851] Lustre: *** cfs_fail_loc=159, val=0*** [ 3038.437986] Lustre: lustre-MDT0000: Client 10b22b4f-ce7d-4368-918f-d54be20faa35 (at 192.168.201.59@tcp) reconnecting [ 3039.798320] Lustre: *** cfs_fail_loc=159, val=0*** [ 3039.815617] Lustre: Skipped 10 previous similar messages [ 3055.296298] Lustre: lustre-MDT0000: Client 10b22b4f-ce7d-4368-918f-d54be20faa35 (at 192.168.201.59@tcp) reconnecting [ 3060.194922] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3060.207283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3060.241383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3060.865157] LustreError: 126885:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3060.884342] LustreError: 126885:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3060.978322] Lustre: server umount lustre-OST0000 complete [ 3065.313614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3065.321470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3068.389207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3068.392336] Lustre: Skipped 1 previous similar message [ 3074.604144] Lustre: server umount lustre-MDT0000 complete [ 3077.573175] LustreError: 126016:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343256 with bad export cookie 17363800710716969619 [ 3077.575685] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3077.579591] LustreError: 126016:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3077.734579] LustreError: 127288:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3077.737535] LustreError: 127288:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3077.856712] Lustre: server umount lustre-MDT0001 complete [ 3086.023100] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 3087.987181] Key type lgssc unregistered [ 3088.227190] LNet: 127820:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3088.234362] LNetError: 127820:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3089.254648] LNet: Removed LNI 192.168.201.159@tcp [ 3089.892312] Key type .llcrypt unregistered [ 3089.895688] Key type ._llcrypt unregistered [ 3108.611122] Key type ._llcrypt registered [ 3108.613659] Key type .llcrypt registered [ 3108.704866] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 3109.226328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3109.327523] alg: No test for adler32 (adler32-zlib) [ 3110.381350] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 3110.544424] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 3112.176856] Key type lgssc registered [ 3112.922941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3119.413559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3119.451570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3120.808356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3124.585295] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3131.371438] LustreError: 129195:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3131.394241] LustreError: 129195:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3131.766187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3132.032280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3135.193281] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3140.067439] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3143.312548] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3149.355139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3149.629256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3154.527378] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3159.591450] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3166.025379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3177.545802] Lustre: *** cfs_fail_loc=159, val=0*** [ 3194.031118] Lustre: lustre-MDT0000: Client db88d6a3-254b-42b4-8346-4febdd6bd3ce (at 192.168.201.59@tcp) reconnecting [ 3195.811149] Lustre: *** cfs_fail_loc=159, val=0*** [ 3195.816280] Lustre: Skipped 1 previous similar message [ 3211.523443] Lustre: lustre-MDT0001: Client db88d6a3-254b-42b4-8346-4febdd6bd3ce (at 192.168.201.59@tcp) reconnecting [ 3212.861680] Lustre: *** cfs_fail_loc=159, val=0*** [ 3212.869381] Lustre: Skipped 2 previous similar messages [ 3228.333277] Lustre: lustre-MDT0001: Client db88d6a3-254b-42b4-8346-4febdd6bd3ce (at 192.168.201.59@tcp) reconnecting [ 3244.830193] Lustre: 131942:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3247.435139] Lustre: *** cfs_fail_loc=159, val=0*** [ 3247.437531] Lustre: Skipped 3 previous similar messages [ 3263.700723] Lustre: lustre-MDT0000: Client 9a2661ee-94d8-4e08-a624-6a6b993249c4 (at 192.168.201.59@tcp) reconnecting [ 3265.205354] Lustre: *** cfs_fail_loc=159, val=0*** [ 3265.207027] Lustre: Skipped 14 previous similar messages [ 3281.609905] Lustre: lustre-MDT0000: Client 9a2661ee-94d8-4e08-a624-6a6b993249c4 (at 192.168.201.59@tcp) reconnecting [ 3282.881603] Lustre: 132239:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3287.013115] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3287.016760] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3287.024068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3289.059520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3289.060394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3294.180964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3294.182932] Lustre: Skipped 2 previous similar messages [ 3299.298562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3299.303743] Lustre: Skipped 1 previous similar message [ 3301.345300] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3301.408605] LustreError: 132336:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3301.521190] Lustre: server umount lustre-OST0000 complete [ 3305.955621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3305.967044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3305.979544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3309.537968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3314.658567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3314.666563] Lustre: Skipped 3 previous similar messages [ 3319.264241] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3319.343629] LustreError: 132541:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3319.347362] LustreError: 132541:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3319.394239] Lustre: server umount lustre-MDT0000 complete [ 3319.785678] LustreError: 129189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3319.802585] LustreError: 129189:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3322.423309] LustreError: 129174:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343501 with bad export cookie 2858099471822244850 [ 3322.428114] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3322.428465] LustreError: 129174:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3322.534199] LustreError: 132743:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3322.544291] LustreError: 132743:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3322.700665] Lustre: server umount lustre-MDT0001 complete [ 3332.049814] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 3334.120512] Key type lgssc unregistered [ 3334.308595] LNet: 133274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3334.312077] LNetError: 133274:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3334.318500] LNet: Removed LNI 192.168.201.159@tcp [ 3334.843941] Key type .llcrypt unregistered [ 3334.847165] Key type ._llcrypt unregistered [ 3355.234682] Key type ._llcrypt registered [ 3355.238494] Key type .llcrypt registered [ 3355.344977] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 3355.998950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3356.075083] alg: No test for adler32 (adler32-zlib) [ 3357.195669] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 3357.494776] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 3359.219510] Key type lgssc registered [ 3359.883675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3365.004864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3365.016656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3366.263628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3368.563905] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3375.080658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3375.462309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3379.585806] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3384.918994] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3388.260574] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3393.263101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3393.507674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3398.314338] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3400.727455] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 3403.198923] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3411.018161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 3419.110951] Lustre: 136943:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3421.187601] Lustre: 136991:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3422.346199] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 3422.945807] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 3425.368146] Lustre: 137146:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 3426.113969] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 3426.820203] Lustre: 137244:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3431.398956] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3431.401048] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3431.411996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3431.416065] Lustre: Skipped 1 previous similar message [ 3435.933199] LustreError: 137341:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3436.109429] Lustre: server umount lustre-OST0000 complete [ 3441.638117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3441.643933] Lustre: Skipped 1 previous similar message [ 3441.650614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3446.364382] LustreError: 137550:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3446.366586] LustreError: 137550:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3446.499951] Lustre: server umount lustre-MDT0000 complete [ 3446.755307] LustreError: 134668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3446.770792] LustreError: 134668:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3450.095851] LustreError: 134649:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343629 with bad export cookie 12158230234529911361 [ 3450.102366] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3450.115079] LustreError: 134649:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3450.290642] LustreError: 137751:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3450.310488] LustreError: 137751:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3450.502220] Lustre: server umount lustre-MDT0001 complete [ 3459.167317] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 3461.005268] Key type lgssc unregistered [ 3461.192653] LNet: 138284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3461.199807] LNetError: 138284:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3461.206845] LNet: Removed LNI 192.168.201.159@tcp [ 3461.725741] Key type .llcrypt unregistered [ 3461.727551] Key type ._llcrypt unregistered [ 3480.894648] Key type ._llcrypt registered [ 3480.895954] Key type .llcrypt registered [ 3480.966190] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 3481.812400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3481.865483] alg: No test for adler32 (adler32-zlib) [ 3482.923343] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 3483.088455] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 3484.736170] Key type lgssc registered [ 3485.647897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3492.812199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3492.845881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3494.657767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3499.636683] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3504.103775] LustreError: 139681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3504.110508] LustreError: 139681:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3508.890551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3509.221595] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3509.559620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3514.281677] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3518.786590] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3522.345552] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3527.919779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3528.235293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3532.968235] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3534.385182] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 3536.940451] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3539.499257] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 3546.793702] Lustre: *** cfs_fail_loc=159, val=0*** [ 3563.190975] Lustre: lustre-MDT0000: Client bb55b5f2-1474-4ec9-907c-68d9860a94f4 (at 192.168.201.59@tcp) reconnecting [ 3580.388110] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3580.391249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3580.403380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3580.436312] Lustre: Skipped 1 previous similar message [ 3580.506676] Lustre: Skipped 1 previous similar message [ 3585.506258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3585.753208] LustreError: 142059:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3585.915823] Lustre: server umount lustre-OST0000 complete [ 3590.628678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3590.629862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3590.643778] Lustre: Skipped 1 previous similar message [ 3590.649030] Lustre: Skipped 2 previous similar messages [ 3595.746229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3595.751611] Lustre: Skipped 1 previous similar message [ 3596.054760] LustreError: 142262:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3596.056584] LustreError: 142262:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3596.108247] Lustre: server umount lustre-MDT0000 complete [ 3598.550318] LustreError: 139660:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343777 with bad export cookie 14134137436357177668 [ 3598.552691] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3598.554822] LustreError: 139660:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3598.654816] LustreError: 142464:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3598.658344] LustreError: 142464:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3598.757911] Lustre: server umount lustre-MDT0001 complete [ 3606.217340] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 3608.576927] Key type lgssc unregistered [ 3608.871822] LNet: 142995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3608.886850] LNetError: 142995:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3609.961907] LNet: Removed LNI 192.168.201.159@tcp [ 3610.659223] Key type .llcrypt unregistered [ 3610.661828] Key type ._llcrypt unregistered [ 3633.185780] Key type ._llcrypt registered [ 3633.187392] Key type .llcrypt registered [ 3633.299877] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 3634.383200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3634.453583] alg: No test for adler32 (adler32-zlib) [ 3635.670572] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 3636.060173] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 3637.760156] Key type lgssc registered [ 3639.047256] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3647.516616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3647.566932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3649.090155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3653.152774] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3659.235302] LustreError: 144393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3659.245703] LustreError: 144393:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3660.152389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3660.368234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3663.348027] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3668.064820] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3671.289581] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3678.154933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3678.507382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3683.919825] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3685.697504] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 3689.668924] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3693.873630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 3711.377596] Lustre: lustre-OST0000: evicting e09a8cd9-5f90-442e-b9b2-5d324395b756 (at 192.168.201.59@tcp) by administrative request [ 3711.383946] Lustre: Skipped 1 previous similar message [ 3740.138724] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3740.142537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3740.146268] Lustre: Skipped 1 previous similar message [ 3740.153231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3744.957244] LustreError: 147176:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3745.069078] Lustre: server umount lustre-OST0000 complete [ 3750.370406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3750.379032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3750.381652] Lustre: Skipped 1 previous similar message [ 3754.530381] LustreError: 147378:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3754.535716] LustreError: 147378:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3754.603091] Lustre: server umount lustre-MDT0000 complete [ 3755.502251] LustreError: 144389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3755.514259] LustreError: 144389:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3757.908652] LustreError: 144374:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343937 with bad export cookie 18214270503325671176 [ 3757.912676] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3757.921724] LustreError: 144374:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3758.086843] LustreError: 147579:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3758.109511] LustreError: 147579:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3758.451466] Lustre: server umount lustre-MDT0001 complete [ 3769.467611] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 3772.764561] Key type lgssc unregistered [ 3773.022751] LNet: 148111:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3773.036643] LNetError: 148111:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3773.054845] LNet: Removed LNI 192.168.201.159@tcp [ 3773.907217] Key type .llcrypt unregistered [ 3773.909552] Key type ._llcrypt unregistered [ 3832.445453] Key type ._llcrypt registered [ 3832.447140] Key type .llcrypt registered [ 3832.516551] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 3833.229737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3833.237662] alg: No test for adler32 (adler32-zlib) [ 3834.214730] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 3834.362251] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 3836.000176] Key type lgssc registered [ 3836.876648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3843.245931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3843.271096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3844.750449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3848.409613] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3855.332189] LustreError: 150557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3855.351073] LustreError: 150557:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3856.702971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3857.495922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3861.728478] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3867.883205] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3872.529839] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3879.215959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3879.560412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3882.653154] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 3886.733255] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3889.794039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 3893.777420] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3903.463405] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3903.474222] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3903.481546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3905.510875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3905.511521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3905.512174] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3905.530729] Lustre: Skipped 1 previous similar message [ 3910.625517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3910.632143] Lustre: Skipped 1 previous similar message [ 3915.749607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3915.755642] Lustre: Skipped 1 previous similar message [ 3917.280158] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3917.346578] LustreError: 152814:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3917.458530] Lustre: server umount lustre-OST0000 complete [ 3923.939410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3923.942905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3923.949389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3925.989993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3927.450815] LustreError: 153019:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3927.453993] LustreError: 153019:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3927.573530] Lustre: server umount lustre-MDT0000 complete [ 3930.683268] LustreError: 150538:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344110 with bad export cookie 15444935108369147894 [ 3930.686780] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3930.694487] LustreError: 150538:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3930.826647] LustreError: 153221:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3930.830208] LustreError: 153221:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3931.030715] Lustre: server umount lustre-MDT0001 complete [ 3939.821767] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 3941.979970] Key type lgssc unregistered [ 3942.233399] LNet: 153753:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3942.249584] LNetError: 153753:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3942.277409] LNet: Removed LNI 192.168.201.159@tcp [ 3943.016495] Key type .llcrypt unregistered [ 3943.017949] Key type ._llcrypt unregistered [ 3965.421501] Key type ._llcrypt registered [ 3965.423036] Key type .llcrypt registered [ 3965.525345] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 3966.436721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3966.463298] alg: No test for adler32 (adler32-zlib) [ 3967.457694] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 3967.677634] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 3969.320194] Key type lgssc registered [ 3970.131098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3975.371590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3981.239559] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 20:49:19 (1763344159) [ 3993.695736] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 4001.483779] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 4008.705785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4013.438078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4017.613734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4023.127952] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4029.209590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4029.263888] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4029.282891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4030.536631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4030.582561] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4030.635423] Lustre: lustre-MDT0000: new disk, initializing [ 4030.780735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4030.806440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4034.390315] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 4042.771308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4042.845719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4042.910298] Lustre: 157974:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4042.972587] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4042.975431] Lustre: Skipped 1 previous similar message [ 4043.043166] Lustre: lustre-MDT0001: new disk, initializing [ 4043.131795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4043.167367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4043.173727] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4046.391161] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 4053.171116] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4056.338880] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4062.571707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4062.644432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4062.862824] Lustre: lustre-OST0000: new disk, initializing [ 4062.875589] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4062.965143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4068.146259] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 4072.014551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4072.047705] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4072.147990] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4074.931473] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4087.785856] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4087.786035] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4087.807636] LustreError: Skipped 1 previous similar message [ 4087.810630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4087.833512] Lustre: Skipped 1 previous similar message [ 4090.757225] LustreError: 160563:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4090.874756] Lustre: server umount lustre-OST0000 complete [ 4098.019054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4098.039853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4098.047735] Lustre: Skipped 2 previous similar messages [ 4101.734477] LustreError: 160764:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4101.737837] LustreError: 160764:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4101.945758] Lustre: server umount lustre-MDT0000 complete [ 4107.276046] LustreError: 157965:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344286 with bad export cookie 15584104947558817439 [ 4107.291577] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4107.329318] LustreError: 157965:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4107.623620] LustreError: 160966:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4107.626729] LustreError: 160966:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4107.943468] Lustre: server umount lustre-MDT0001 complete [ 4122.466790] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 4126.278711] Key type lgssc unregistered [ 4126.530791] LNet: 161498:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4126.538884] LNetError: 161498:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4126.555953] LNet: Removed LNI 192.168.201.159@tcp [ 4127.659081] Key type .llcrypt unregistered [ 4127.660427] Key type ._llcrypt unregistered [ 4153.363114] Key type ._llcrypt registered [ 4153.366087] Key type .llcrypt registered [ 4153.508646] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 4154.530924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4154.756624] alg: No test for adler32 (adler32-zlib) [ 4155.955839] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 4156.190267] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 4157.888256] Key type lgssc registered [ 4159.213793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4167.994137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4168.029332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4169.573463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4173.306706] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 4178.914894] LustreError: 162894:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4178.931038] LustreError: 162894:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4181.438238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4181.875505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4185.758316] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 4191.569369] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4195.576160] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4202.203562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4202.674435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4208.034507] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 4212.579427] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4213.959754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4222.603054] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4222.619294] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4222.630274] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4222.635865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4222.648248] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4223.441304] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4223.466456] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4223.485459] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4223.490990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4223.504615] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4224.463932] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4224.495343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4224.509651] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4224.513312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4224.531475] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4225.350900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4226.121191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4226.822178] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4226.825821] Lustre: Skipped 2 previous similar messages [ 4226.848594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4226.861834] Lustre: Skipped 2 previous similar messages [ 4226.865929] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4226.869710] Lustre: Skipped 2 previous similar messages [ 4226.879373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4226.885933] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4226.888176] Lustre: Skipped 2 previous similar messages [ 4227.764760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4228.583264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4229.434102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4230.226387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4231.141299] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4231.144015] Lustre: Skipped 4 previous similar messages [ 4231.151810] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4231.157428] Lustre: Skipped 4 previous similar messages [ 4231.160486] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4231.171625] Lustre: Skipped 4 previous similar messages [ 4231.177271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4231.185667] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4231.188711] Lustre: Skipped 4 previous similar messages [ 4232.178514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4233.002421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4233.807731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4234.810307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4235.770181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4236.702181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4237.605393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4238.594385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4239.462879] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4239.473967] Lustre: Skipped 8 previous similar messages [ 4239.495941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4239.502982] Lustre: Skipped 8 previous similar messages [ 4239.506819] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4239.511197] Lustre: Skipped 8 previous similar messages [ 4239.513547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4239.524157] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4239.529817] Lustre: Skipped 8 previous similar messages [ 4240.393816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4241.364541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4242.089357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4242.929465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4243.821469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4244.745575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4245.660656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4246.589071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4247.492613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4248.507398] LustreError: 162914:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4248.519328] LustreError: 162914:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4248.532094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4249.416659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4250.325350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4251.215921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4252.233274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4253.135576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4254.115376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4255.263645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4256.460196] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4256.463046] Lustre: Skipped 17 previous similar messages [ 4256.476884] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4256.491653] Lustre: Skipped 17 previous similar messages [ 4256.501612] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4256.516438] Lustre: Skipped 17 previous similar messages [ 4256.526081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4256.537392] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4256.541262] Lustre: Skipped 17 previous similar messages [ 4257.464066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4258.527932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4259.411419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4260.399314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4261.362274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4262.265289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4263.334600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4264.160066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4264.846928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4265.642737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4266.499906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4267.376638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4268.100462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4268.935864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4269.697397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4270.415492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4271.139322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4271.995557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4272.770432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4273.740071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4274.743913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4275.775290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4276.921951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4277.771188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4278.729822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4279.892431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4281.027607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4282.257893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4283.545041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4284.449486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4285.891651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4286.933441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4288.249880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4289.421615] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4289.434649] Lustre: Skipped 33 previous similar messages [ 4289.462116] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4289.472300] Lustre: Skipped 33 previous similar messages [ 4289.478090] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4289.481266] Lustre: Skipped 33 previous similar messages [ 4289.484586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4289.495042] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4289.502711] Lustre: Skipped 33 previous similar messages [ 4290.457481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4291.537570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4292.450363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4293.627082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4294.567124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4295.641018] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4296.670869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4297.667508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4299.048579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4300.045647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4301.126576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4302.061317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4303.052210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4304.159734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4304.998732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4305.846635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4306.663724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4307.637843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4308.830314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4309.686819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4310.635430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4311.486070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4312.454241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4313.248723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4314.045484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4314.900128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4315.845300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4316.755540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4317.560600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4318.334893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4319.047113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4319.832714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4320.755679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4321.366174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4322.014343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4322.711547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4323.712403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4324.510245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4325.513957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4326.276355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4327.014430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4327.763851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4328.567972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4329.341793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4330.104181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4330.747840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4331.553942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4332.399073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4333.163993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4333.965291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4334.776475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4335.577278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4336.303200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4337.119301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4337.876561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4338.740710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4339.563855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4340.525363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4341.450136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4342.303987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4343.166261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4343.919776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4344.695472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4345.490929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4346.368289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4347.326659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4348.151461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4348.891458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4349.775304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4350.493660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4351.369496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4352.100478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4352.944701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4353.770840] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4353.778946] Lustre: Skipped 73 previous similar messages [ 4353.800968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4353.809186] Lustre: Skipped 73 previous similar messages [ 4353.813380] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4353.816573] Lustre: Skipped 73 previous similar messages [ 4353.820708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4353.827161] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4353.830704] Lustre: Skipped 73 previous similar messages [ 4354.713121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4355.579675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4356.438298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4357.345814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4358.239354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4359.107717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4360.115923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4360.918631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4361.654484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4362.434089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4363.325164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4364.231678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4365.260793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4366.193124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4367.194943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4368.097570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4368.943965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4369.890512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4370.837366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4371.693564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4372.537203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4373.338842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4374.207431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4375.053510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4375.815572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4376.657418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4377.500708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4378.222475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4379.156196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4380.040698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4380.857952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4381.819423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4382.680306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4383.648539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4384.700770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4385.580487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4386.383424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4387.369873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4388.240447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4388.984940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4389.890559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4390.668682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4391.507626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4392.462888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4393.238199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4394.055711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4394.779126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4395.547448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4396.241187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4397.040211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4398.030198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4398.876960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4399.765411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4400.667694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4401.606322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4402.355370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4403.127303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4403.950482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4404.849697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4405.761533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4406.610977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4407.604239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4408.729811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4409.713662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4410.761186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4411.704377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4412.558245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4413.414672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4414.423038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4415.458425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4416.352695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4417.448315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4418.254969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4419.219462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4420.267713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4421.155366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4421.979524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4422.915374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4423.886678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4424.660406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4425.528722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4426.351810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4427.069181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4427.923100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4428.706430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4429.597045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4430.383276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4431.123626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4431.849285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4432.548862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4433.203721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4433.901575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4434.490409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4435.150752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4435.829943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4436.536074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4437.195150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4437.866967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4438.524882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4439.360593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4439.996531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4440.607965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4441.266592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4441.933367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4442.752950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4443.417749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4444.153044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4444.854082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4445.461586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4446.105305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4446.704625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4447.293992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4447.928884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4448.622948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4449.336445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4450.067745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4450.863068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4451.686836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4452.411250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4453.176531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4453.941093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4454.765828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4455.568673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4456.327820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4457.213379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4458.071322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4459.009527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4459.751656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4460.641143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4461.415261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4462.374275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4463.253414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4464.331079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4464.977633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4465.677709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4466.329046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4466.964918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4467.664910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4468.314317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4469.062861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4469.751871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4470.376631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4471.182694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4471.950730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4472.805824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4473.599213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4474.480670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4475.324359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4476.312961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4477.138431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4477.883637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4478.680775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4479.490762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4480.234453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4481.206187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4482.032615] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4482.036386] Lustre: Skipped 155 previous similar messages [ 4482.057805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4482.079592] Lustre: Skipped 155 previous similar messages [ 4482.082974] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4482.085950] Lustre: Skipped 155 previous similar messages [ 4482.098510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4482.111581] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4482.114457] Lustre: Skipped 155 previous similar messages [ 4483.034771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4483.815848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4484.885459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4485.779940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4486.698304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4487.537866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4488.419196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4489.207507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4490.047479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4490.909725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4491.808600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4492.533927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4493.567227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4494.469822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4495.641957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4496.620922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4497.550664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4498.475172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4499.613029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4500.929271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4502.086385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4503.431595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4504.802201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4505.802068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4506.845955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4508.060665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4509.020890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4509.880754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4510.762691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4511.621313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4512.564416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4513.563260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4514.497732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4515.376195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4516.420025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4517.397869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4518.272765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4519.122838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4520.125538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4521.149969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4522.185745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4523.060564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4524.262682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4525.373839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4526.464869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4527.446083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4528.425629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4529.406391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4530.376095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4531.479175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4532.285776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4533.177763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4534.067540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4534.909329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4535.672642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4536.634471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4537.594789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4538.620886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4539.696067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4540.526548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4541.397832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4542.278858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4543.278944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4544.126599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4544.934599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4545.777347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4546.535854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4547.409779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4548.358444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4549.226423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4550.207870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4551.092854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4552.095931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4553.058916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4553.989742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4554.716138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4555.518298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4556.351199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4557.309076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4558.075366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4558.875857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4559.733388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4560.728566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4561.580615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4562.509235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4563.380722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4564.253619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4565.036349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4566.069614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4566.844491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4567.588076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4568.371806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4569.117518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4569.788693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4570.489887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4571.224674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4571.919333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4572.531776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4573.286351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4574.027718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4574.782635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4575.549863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4576.325508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4577.151993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4578.088391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4579.064379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4579.941890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4580.684557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4581.385645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4582.122851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4583.056274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4583.832502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4584.546490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4585.383328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4586.156762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4586.835523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4587.565581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4588.333424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4589.141859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4589.941539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4590.827922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4591.807253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4592.738640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4593.520851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4594.294582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4595.113256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4595.890132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4596.694693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4597.449398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4598.101278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4598.811733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4599.570741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4600.234744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4600.987940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4601.719225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4602.539440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4603.325835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4604.226542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4605.088682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4605.904892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4606.658459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4607.564488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4608.334058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4609.191301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4609.915811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4610.719102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4611.368241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4612.074453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4612.712611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4613.497977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4614.307703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4615.077707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4615.761248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4616.479911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4617.121823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4617.759317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4618.370365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4618.987412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4619.744795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4620.373366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4621.211048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4622.051471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4622.878976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4623.806371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4624.637775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4625.336611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4626.182849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4626.957770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4627.727357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4628.431736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4629.438984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4630.266321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4630.897763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4631.678920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4632.429392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4633.188590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4633.939500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4634.654762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4635.515877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4636.267957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4636.989596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4637.662738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4638.488902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4639.258936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4639.949483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4640.669193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4641.634132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4642.442949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4643.167193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4644.171862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4644.996236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4645.955524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4646.929044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4647.806084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4648.661917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4649.574638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4650.468850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4651.475570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4652.289364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4653.159726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4653.867533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4654.560751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4655.283097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4655.989738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4656.886660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4657.797991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4658.612500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4659.615366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4660.444418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4661.257913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4662.048349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4662.841995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4663.591370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4664.328478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4665.071158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4665.878380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4666.685135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4667.564144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4668.520075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4669.595260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4670.336729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4671.202184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4671.877248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4672.732531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4673.505090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4674.198325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4674.888645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4675.580840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4676.340834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4677.022337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4677.792234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4678.515557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4679.211535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4679.891347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4680.495802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4681.110239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4681.695788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4682.566678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4683.448974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4684.331697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4685.183558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4685.929426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4686.736181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4687.442315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4688.242693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4689.046423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4689.894025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4690.614651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4691.424532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4692.201801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4693.024585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4693.717830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4694.376320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4695.049807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4695.710582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4696.453973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4697.144468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4697.962996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4698.738093] LustreError: 162914:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4698.748935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4698.761094] LustreError: 162914:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4699.539764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4700.306054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4700.962309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4701.719128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4702.415897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4703.156956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4703.840472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4704.473081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4705.107716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4784.191680] LustreError: 194689:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4784.361561] Lustre: server umount lustre-OST0000 complete [ 4789.734104] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4789.735127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4789.743860] Lustre: Skipped 269 previous similar messages [ 4794.853208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4794.856085] Lustre: Skipped 2 previous similar messages [ 4797.946552] LustreError: 194890:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4797.956215] LustreError: 194890:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4798.026684] Lustre: server umount lustre-MDT0000 complete [ 4799.906181] Lustre: 162204:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763344963/real 1763344963] req@0000000072334056 x1849000551116544/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763344979 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4799.942436] LustreError: 162895:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4799.967623] LustreError: 162895:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4800.992132] Lustre: 162203:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763344964/real 1763344964] req@000000007ab92f83 x1849000551116800/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1763344980 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4801.648459] LustreError: 162874:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344980 with bad export cookie 10547468731285446953 [ 4801.663275] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4801.790910] LustreError: 195092:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4801.811164] LustreError: 195092:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4801.965114] Lustre: server umount lustre-MDT0001 complete [ 4809.170125] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 4811.267564] Key type lgssc unregistered [ 4811.488867] LNet: 195624:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4811.497160] LNetError: 195624:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4811.508185] LNet: Removed LNI 192.168.201.159@tcp [ 4812.222500] Key type .llcrypt unregistered [ 4812.225274] Key type ._llcrypt unregistered [ 4829.846818] Key type ._llcrypt registered [ 4829.848775] Key type .llcrypt registered [ 4829.957501] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 4830.682837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4830.719819] alg: No test for adler32 (adler32-zlib) [ 4831.765217] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 4831.972657] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 4833.664166] Key type lgssc registered [ 4834.607091] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4840.623853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4840.649749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4842.186967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4845.754240] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 4851.682594] LustreError: 196999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4851.701623] LustreError: 196999:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4852.915123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4853.268728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4856.715416] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 4861.989803] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4865.611127] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4872.619836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4872.896519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4878.294137] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 4883.741634] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4887.215299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9780 to 0x280000401:9825) [ 4891.976023] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4893.371431] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4893.378284] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4893.382472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4893.388447] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4893.388690] LustreError: 197018:0:(osp_precreate.c:1263:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 4898.791559] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4898.794368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4898.801544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4898.806854] Lustre: Skipped 1 previous similar message [ 4899.496293] Lustre: lustre-OST0000: Not available for connect from 192.168.201.59@tcp (stopping) [ 4899.500711] Lustre: Skipped 1 previous similar message [ 4900.920148] LustreError: 199398:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4901.015900] Lustre: server umount lustre-OST0000 complete [ 4913.448177] LustreError: 197018:0:(osp_precreate.c:1263:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 4932.887102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4933.171631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4937.988446] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 4939.304814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4939.312968] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4939.313906] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4939.325922] Lustre: Skipped 1 previous similar message [ 4941.771041] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4942.884117] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4945.914158] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff93dd10dc6800.ost_server_uuid 50 [ 4947.081135] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff93dd10dc6800.ost_server_uuid in FULL state after 0 sec [ 4954.595419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4954.599251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4954.601656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4954.608251] Lustre: Skipped 1 previous similar message [ 4955.707772] LustreError: 200719:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4955.717788] LustreError: 200719:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4955.800210] Lustre: server umount lustre-OST0000 complete [ 4959.717339] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4959.718765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4959.728373] Lustre: Skipped 1 previous similar message [ 4959.744996] Lustre: Skipped 2 previous similar messages [ 4964.761332] LustreError: 200921:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4964.763742] LustreError: 200921:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4964.831715] Lustre: server umount lustre-MDT0000 complete [ 4964.851053] LustreError: 196995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4964.874151] LustreError: 196995:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4967.774083] LustreError: 196980:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345147 with bad export cookie 15052629600097739320 [ 4967.782058] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4967.783449] LustreError: 196980:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4967.948669] LustreError: 201122:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4967.956937] LustreError: 201122:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4968.116741] Lustre: server umount lustre-MDT0001 complete [ 4975.915449] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 4978.289827] Key type lgssc unregistered [ 4978.545958] LNet: 201653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4978.560288] LNetError: 201653:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4979.626272] LNet: Removed LNI 192.168.201.159@tcp [ 4980.408356] Key type .llcrypt unregistered [ 4980.410747] Key type ._llcrypt unregistered [ 5004.303078] Key type ._llcrypt registered [ 5004.304575] Key type .llcrypt registered [ 5004.385792] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 5005.579850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5005.613949] alg: No test for adler32 (adler32-zlib) [ 5006.660984] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 5006.910224] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 5008.648137] Key type lgssc registered [ 5009.899645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5020.151559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5020.163706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5021.559395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5024.903589] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 5026.793717] LustreError: 203342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5026.823577] LustreError: 203342:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5030.883778] LustreError: 203342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5033.057092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5033.493466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5037.092587] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 5040.165889] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5046.978281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5047.226407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5052.572335] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 5060.521216] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5060.583591] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5060.732205] Lustre: lustre-OST0001: new disk, initializing [ 5060.735826] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5060.818850] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5065.840706] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 5065.970547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9827 to 0x280000401:9857) [ 5066.304453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5066.314207] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5066.403163] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5075.950770] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5083.341880] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5100.004576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5100.009398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5100.020178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5100.833929] LustreError: 206699:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5100.942978] Lustre: server umount lustre-MDT0000 complete [ 5102.566130] LustreError: 203342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5102.570221] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5102.606200] Lustre: Skipped 2 previous similar messages [ 5103.933368] LustreError: 203322:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345283 with bad export cookie 1151730348934415948 [ 5103.937635] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5103.938288] LustreError: 203322:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5104.063942] LustreError: 206900:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5104.067764] LustreError: 206900:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5104.218526] Lustre: server umount lustre-MDT0001 complete [ 5116.929100] LustreError: 207100:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5116.932807] LustreError: 207100:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5116.974799] Lustre: server umount lustre-OST0000 complete [ 5130.224130] LustreError: 207303:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5130.233886] LustreError: 207303:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5130.382657] Lustre: server umount lustre-OST0001 complete [ 5133.857238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5133.981249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5135.272976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5135.434419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5136.775893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5136.915346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5138.184291] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5138.291209] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5150.368897] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 5159.369341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5159.724748] LustreError: 208714:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5159.737110] LustreError: 208714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5159.796903] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5162.745623] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 5165.033755] LustreError: 208714:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5168.977372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5169.220727] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5172.340157] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 5174.783675] Lustre: 209823:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5180.402455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5184.811355] LustreError: 210179:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5184.813700] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9859 to 0x280000401:9889) [ 5185.269532] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 5191.616755] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5191.789797] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5191.793725] Lustre: Skipped 1 previous similar message [ 5193.122791] Lustre: mylustre-OST0001: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5196.703483] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 5203.637331] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5208.024090] Lustre: 211671:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5224.941803] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5224.944188] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5224.959439] Lustre: Skipped 2 previous similar messages [ 5224.966646] Lustre: Skipped 3 previous similar messages [ 5229.368401] LustreError: 211977:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5229.377108] LustreError: 211977:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5229.474687] Lustre: server umount mylustre-MDT0000 complete [ 5230.055820] LustreError: 208707:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5230.072908] LustreError: 208707:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5233.362200] LustreError: 208688:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345412 with bad export cookie 1151730348934418755 [ 5233.365526] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5233.411892] LustreError: 208688:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5233.773278] Lustre: server umount mylustre-MDT0001 complete [ 5247.681223] LustreError: 212382:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 5247.687037] LustreError: 212382:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 5247.719358] Lustre: server umount mylustre-OST0000 complete [ 5260.726073] Lustre: server umount mylustre-OST0001 complete [ 5263.712629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5263.827907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5265.040663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5265.136735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5266.310245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5266.406989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5267.521199] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5267.644073] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5279.166034] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 5287.999034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5288.476463] LustreError: 213995:0:(ldlm_lib.c:1179:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5288.549204] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5291.847982] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 5298.846277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5302.906545] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 5305.884722] Lustre: 215103:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5312.302126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5314.669062] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9891 to 0x280000401:9921) [ 5318.044933] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 5326.564476] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5328.772760] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 5332.462584] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 5338.911683] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5341.794877] Lustre: 216951:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5352.421229] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5352.423110] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 5352.432439] Lustre: Skipped 3 previous similar messages [ 5352.439971] Lustre: Skipped 3 previous similar messages [ 5357.540915] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 5357.560813] Lustre: Skipped 3 previous similar messages [ 5357.947955] LustreError: 217200:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5357.956985] LustreError: 217200:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5358.053823] Lustre: server umount tfs-MDT0000 complete [ 5360.644883] LustreError: 213969:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345539 with bad export cookie 1151730348934420645 [ 5360.651801] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5360.657717] LustreError: 213969:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5360.881623] Lustre: server umount tfs-MDT0001 complete [ 5374.261165] Lustre: server umount tfs-OST0000 complete [ 5387.718191] Lustre: server umount tfs-OST0001 complete [ 5390.234468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5390.280297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5391.087534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5391.137640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5391.924967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5392.009082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5392.941531] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5393.032649] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5405.372232] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 5416.031714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5416.864056] LustreError: 219214:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5416.871481] LustreError: 219214:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 5417.004624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5417.011323] Lustre: Skipped 3 previous similar messages [ 5421.167063] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 5430.349803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5435.267818] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 5438.719593] Lustre: 220326:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5444.970377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5450.374459] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 5457.248103] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5462.019332] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 5462.330348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9891 to 0x280000401:9953) [ 5462.338362] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 5468.494873] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5471.476122] Lustre: 222173:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5481.576696] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 21:14:19 (1763345659) [ 5497.314955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5497.318938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5497.339848] Lustre: Skipped 1 previous similar message [ 5497.346262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5503.146729] LustreError: 222540:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5503.156808] LustreError: 222540:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 5503.272744] Lustre: server umount lustre-MDT0000 complete [ 5507.692785] LustreError: 219189:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345687 with bad export cookie 1151730348934422311 [ 5507.699662] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5507.709792] LustreError: 219189:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5508.578559] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5508.590669] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5508.601479] Lustre: Skipped 4 previous similar messages [ 5508.612878] Lustre: Skipped 5 previous similar messages [ 5514.067745] Lustre: server umount lustre-MDT0001 complete [ 5523.698027] Lustre: server umount lustre-OST0000 complete [ 5541.344117] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5541.538872] Lustre: server umount lustre-OST0001 complete [ 5547.465820] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 5555.524374] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 5564.897610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5571.180798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5576.634403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5582.465195] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5588.382303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5588.443798] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 5588.443798] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 5588.443798] [ 5588.474917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 5588.628762] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5588.660578] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5588.754657] Lustre: lustre-MDT0000: new disk, initializing [ 5588.899909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5588.907980] Lustre: Skipped 3 previous similar messages [ 5588.935297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5592.526823] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 5601.774473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5601.844407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5601.922844] Lustre: 225336:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5601.972877] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5601.975508] Lustre: Skipped 1 previous similar message [ 5602.044752] Lustre: lustre-MDT0001: new disk, initializing [ 5602.160556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5602.173532] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5606.301928] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 5616.670270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5616.732793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5616.912761] Lustre: lustre-OST0000: new disk, initializing [ 5616.917721] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5618.186276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5618.190932] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5618.297827] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5621.956729] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 5630.776318] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5630.834293] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5630.952307] Lustre: lustre-OST0001: new disk, initializing [ 5630.954779] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5632.166487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5632.171778] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5632.260150] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5636.383757] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 5647.771684] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 21:17:04 (1763345824) [ 5661.135663] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 21:17:18 (1763345838) [ 5678.572084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5678.576367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5678.578470] Lustre: Skipped 1 previous similar message [ 5678.585812] Lustre: Skipped 3 previous similar messages [ 5680.641088] LustreError: 228759:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5680.649181] LustreError: 228759:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 5680.706518] Lustre: server umount lustre-OST0000 complete [ 5683.688791] LustreError: 226870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5683.705953] LustreError: 226870:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 5684.197668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5690.238984] Lustre: server umount lustre-MDT0000 complete [ 5693.491329] LustreError: 225328:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345872 with bad export cookie 1151730348934423865 [ 5693.496688] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5693.498042] LustreError: 225328:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5699.796865] Lustre: server umount lustre-MDT0001 complete [ 5707.037598] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 5730.272342] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5730.457687] Lustre: server umount lustre-OST0001 complete [ 5736.084354] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 5751.357248] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 5761.719822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5768.654294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5775.182428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5781.311655] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5786.864996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5786.928300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5787.146928] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5787.168392] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5787.233738] Lustre: lustre-MDT0000: new disk, initializing [ 5787.322619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5790.860802] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 5802.172150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5802.282958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5802.417782] Lustre: 232249:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5802.713797] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5806.867453] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 5814.328721] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5817.647659] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5824.168993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5824.221571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5824.486563] Lustre: lustre-OST0000: new disk, initializing [ 5824.490446] Lustre: Skipped 1 previous similar message [ 5824.501623] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5824.511200] Lustre: Skipped 2 previous similar messages [ 5825.710676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5825.714275] Lustre: Skipped 1 previous similar message [ 5825.741893] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5825.877770] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5830.603807] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 5838.325635] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5846.498852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5846.508230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5846.534416] Lustre: Skipped 4 previous similar messages [ 5846.540314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5846.545304] Lustre: Skipped 7 previous similar messages [ 5850.502150] LustreError: 234691:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5850.521506] LustreError: 234691:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 5850.800486] Lustre: server umount lustre-OST0000 complete [ 5861.424146] Lustre: server umount lustre-MDT0000 complete [ 5865.689291] LustreError: 232242:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346045 with bad export cookie 1151730348934425335 [ 5865.696720] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5865.746060] LustreError: 232242:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5866.424367] Lustre: server umount lustre-MDT0001 complete [ 5878.369326] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 5881.062638] Key type lgssc unregistered [ 5881.324736] LNet: 235628:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5881.341550] LNetError: 235628:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5881.349509] LNet: Removed LNI 192.168.201.159@tcp [ 5882.138185] Key type .llcrypt unregistered [ 5882.143985] Key type ._llcrypt unregistered [ 5905.884693] Key type ._llcrypt registered [ 5905.886264] Key type .llcrypt registered [ 5905.970240] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 5907.121518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5907.157740] alg: No test for adler32 (adler32-zlib) [ 5908.214876] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 5908.405931] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 5910.056442] Key type lgssc registered [ 5911.136091] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5918.665505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5918.689341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5920.171100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5924.053682] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 5930.469629] LustreError: 237078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5930.498827] LustreError: 237078:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5932.555202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5932.877978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5937.799380] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 5944.309097] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5949.343638] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5958.799550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5959.248387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5965.428498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5965.768909] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 5971.612950] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5972.882381] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 5976.034643] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5976.056481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5978.152811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5983.214152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5983.216985] Lustre: Skipped 1 previous similar message [ 5988.327961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5988.330731] Lustre: Skipped 1 previous similar message [ 5989.344173] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5989.394151] LustreError: 239372:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5989.578079] Lustre: server umount lustre-OST0000 complete [ 5994.475213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5994.480880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5994.496189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5994.498607] Lustre: Skipped 1 previous similar message [ 5998.561730] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6003.681836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6003.688202] Lustre: Skipped 3 previous similar messages [ 6008.288139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6008.410941] LustreError: 239581:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6008.416484] LustreError: 239581:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6008.553105] Lustre: server umount lustre-MDT0000 complete [ 6008.808408] LustreError: 238939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6008.825032] LustreError: 238939:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6013.119026] LustreError: 239275:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346192 with bad export cookie 17721205737338839102 [ 6013.127077] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6013.130806] LustreError: 239275:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6013.273193] LustreError: 239785:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6013.275521] LustreError: 239785:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6013.435443] Lustre: server umount lustre-MDT0001 complete [ 6024.391735] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 6027.772862] Key type lgssc unregistered [ 6028.076090] LNet: 240317:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6028.095282] LNetError: 240317:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6028.116259] LNet: Removed LNI 192.168.201.159@tcp [ 6029.001650] Key type .llcrypt unregistered [ 6029.003530] Key type ._llcrypt unregistered [ 6046.132121] Key type ._llcrypt registered [ 6046.139284] Key type .llcrypt registered [ 6046.276241] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 6047.505836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6047.589943] alg: No test for adler32 (adler32-zlib) [ 6048.891293] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 6049.157921] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 6050.824177] Key type lgssc registered [ 6052.450371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6061.783573] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6061.833978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6077.408337] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6082.528227] LustreError: 241491:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.159@tcp: failed processing log, type 4: rc = -110 [ 6109.152193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6114.421911] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6119.724556] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6122.685214] LustreError: 242158:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 6122.953549] Lustre: server umount lustre-OST0000 complete [ 6142.270198] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 6145.459488] Key type lgssc unregistered [ 6145.782516] LNet: 242995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6145.801515] LNetError: 242995:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6145.827615] LNet: Removed LNI 192.168.201.159@tcp [ 6146.725751] Key type .llcrypt unregistered [ 6146.727400] Key type ._llcrypt unregistered [ 6187.039900] Key type ._llcrypt registered [ 6187.044143] Key type .llcrypt registered [ 6187.181953] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 6187.794453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6187.817548] alg: No test for adler32 (adler32-zlib) [ 6188.838110] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 6189.091907] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 6190.768197] Key type lgssc registered [ 6191.810488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6204.103500] loop0: detected capacity change from 0 to 204800000 [ 6204.303639] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6204.316769] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6204.564765] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6204.629734] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6205.653694] loop0: detected capacity change from 0 to 204800000 [ 6205.866975] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6205.884887] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6206.231820] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6206.296393] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6207.430549] loop0: detected capacity change from 0 to 204800000 [ 6207.610382] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6207.623185] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6207.794963] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6207.946572] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6209.046253] loop0: detected capacity change from 0 to 204800000 [ 6209.291612] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6209.317051] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6209.436374] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6209.529918] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6210.610690] loop0: detected capacity change from 0 to 204800000 [ 6210.645756] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6215.009765] loop0: detected capacity change from 0 to 204800000 [ 6215.046903] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6219.761100] loop0: detected capacity change from 0 to 204800000 [ 6219.789643] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6223.951953] loop0: detected capacity change from 0 to 204800000 [ 6223.995360] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6227.968756] loop0: detected capacity change from 0 to 204800000 [ 6228.081343] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6228.133920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6228.158176] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6228.216992] Lustre: 246401:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6228.225178] Lustre: 246401:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 6228.233477] Lustre: 246401:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 6228.246781] Lustre: 246401:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 6228.254146] Lustre: 246401:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 6228.262913] Lustre: 246401:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 6228.274462] Lustre: 246401:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 6228.289477] Lustre: 246401:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 6228.300226] Lustre: 246401:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 6228.320672] Lustre: 246401:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 6228.333970] Lustre: 246401:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 6228.369582] Lustre: 246401:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 6228.382310] Lustre: 246401:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 6228.392902] Lustre: 246401:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 6233.478038] LustreError: 246667:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6233.578360] Lustre: server umount MGS complete [ 6234.447326] loop0: detected capacity change from 0 to 204800000 [ 6234.588063] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6235.000634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6235.043080] Lustre: lustre-MDT0000: Aborting client recovery [ 6235.832761] loop1: detected capacity change from 0 to 204800000 [ 6236.020976] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6236.082891] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6236.128145] Lustre: 246836:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6236.134322] Lustre: 246836:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 6236.140421] Lustre: 246836:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 6236.150214] Lustre: 246836:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 6236.159707] Lustre: 246836:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 6236.165357] Lustre: 246836:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 6236.172493] Lustre: 246836:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 6236.184980] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6236.339541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6236.398265] Lustre: lustre-MDT0001: Aborting client recovery [ 6237.242701] loop2: detected capacity change from 0 to 204800000 [ 6237.360823] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6237.419997] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6237.464347] Lustre: 246918:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6237.469484] Lustre: 246918:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 6237.475900] Lustre: 246918:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 6237.483159] Lustre: 246918:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 6237.488647] Lustre: 246918:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 6237.495220] Lustre: 246918:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 6237.501738] Lustre: 246918:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 6237.622183] Lustre: Found index 0 for lustre-OST0000, updating log [ 6237.727480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6237.742253] Lustre: lustre-OST0000: Aborting recovery [ 6238.505703] loop3: detected capacity change from 0 to 204800000 [ 6238.630626] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6238.717178] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6238.744904] Lustre: 247005:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6238.751547] Lustre: 247005:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 6238.758618] Lustre: 247005:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 6238.765466] Lustre: 247005:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 6238.774211] Lustre: 247005:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 6238.781453] Lustre: 247005:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 6238.790991] Lustre: 247005:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 6238.812610] Lustre: Found index 1 for lustre-OST0001, updating log [ 6238.933865] Lustre: lustre-OST0001: Aborting recovery [ 6244.387335] LustreError: 246779:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -108 [ 6245.353463] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 6245.360214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 6245.431034] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 6245.445516] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 6246.376834] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 6246.385469] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 6247.763810] LustreError: 246747:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 384us: rc = -2 [ 6248.265676] LustreError: 247253:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 2427us: rc = -2 [ 6248.273489] LustreError: 247253:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 25 previous similar messages [ 6261.730131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6261.732267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6261.734686] Lustre: Skipped 2 previous similar messages [ 6261.754709] Lustre: Skipped 3 previous similar messages [ 6266.549275] LustreError: 247360:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6266.552256] LustreError: 247360:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6266.851404] LustreError: 246748:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6266.863767] LustreError: 246748:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6268.643540] LustreError: 247360:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 6268.646482] LustreError: 247360:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6268.703655] Lustre: server umount lustre-MDT0000 complete [ 6269.581769] LustreError: 246731:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346448 with bad export cookie 10849582149524744889 [ 6269.592317] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6271.968895] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6271.972348] LustreError: 246744:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6271.973085] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6271.979660] Lustre: Skipped 2 previous similar messages [ 6271.993473] LustreError: 246744:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6274.976755] LustreError: 246744:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6274.980298] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6274.997121] LustreError: 246744:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6275.013500] Lustre: Skipped 2 previous similar messages [ 6275.819213] LustreError: 247412:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 6275.832081] LustreError: 247412:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6276.006538] Lustre: server umount lustre-MDT0001 complete [ 6282.874524] LustreError: 247464:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6282.883098] LustreError: 247464:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6282.956982] Lustre: server umount lustre-OST0000 complete [ 6298.083828] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6298.134685] LustreError: 247514:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6298.147524] LustreError: 247514:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6298.342770] Lustre: server umount lustre-OST0001 complete [ 6305.729711] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 21:28:03 (1763346483) [ 6330.370086] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 6338.327487] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 6347.165940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6352.984590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6358.579872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6365.148386] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6372.673606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6372.768284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6373.040226] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6373.083644] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6373.182600] Lustre: lustre-MDT0000: new disk, initializing [ 6373.296875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6373.304353] Lustre: Skipped 1 previous similar message [ 6373.324477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6377.410493] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6387.619941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6387.670093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6387.768697] Lustre: 251067:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6387.831843] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6387.850700] Lustre: Skipped 1 previous similar message [ 6387.911926] Lustre: lustre-MDT0001: new disk, initializing [ 6387.986528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6388.023495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6388.034281] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6391.527809] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6398.276030] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6401.525506] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6409.282598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6409.356520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6409.654873] Lustre: lustre-OST0000: new disk, initializing [ 6409.658074] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6409.734123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6415.663683] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6417.491405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6417.505845] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6417.667224] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6423.368411] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6428.012602] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 6437.064194] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 6444.362936] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 6494.691233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6494.694590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6494.711773] Lustre: Skipped 1 previous similar message [ 6494.722165] Lustre: Skipped 1 previous similar message [ 6495.674222] LustreError: 254586:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6495.679024] LustreError: 254586:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6495.762317] Lustre: server umount lustre-OST0000 complete [ 6499.810070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6499.813732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6499.817790] Lustre: Skipped 1 previous similar message [ 6504.913966] Lustre: server umount lustre-MDT0000 complete [ 6504.930302] LustreError: 251075:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6504.943486] LustreError: 251075:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6507.760091] LustreError: 254260:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346687 with bad export cookie 10849582149524747591 [ 6507.762161] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6507.770349] LustreError: 254260:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6508.067731] Lustre: server umount lustre-MDT0001 complete [ 6515.240693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6518.291792] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6522.240930] Lustre: server umount MGS complete [ 6530.957504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6531.337582] LustreError: 256556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6531.366406] LustreError: 256568:0:(qmt_pool.c:1523:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 6531.414806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6534.336661] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6540.771646] LustreError: 256556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6540.790244] LustreError: 256556:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6541.039709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6544.352841] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6548.204883] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6550.782532] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6555.947779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6561.580252] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6566.381960] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6588.385965] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6588.393518] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6588.403479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6588.407681] Lustre: Skipped 1 previous similar message [ 6592.490306] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6593.554918] LustreError: 259209:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6593.570407] LustreError: 259209:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 6593.746547] Lustre: server umount lustre-OST0000 complete [ 6597.603724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6597.611744] Lustre: Skipped 1 previous similar message [ 6597.619779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6603.754309] Lustre: server umount lustre-MDT0000 complete [ 6606.573846] LustreError: 256537:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346785 with bad export cookie 10849582149524749439 [ 6606.577373] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6606.583541] LustreError: 256537:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6606.955487] Lustre: server umount lustre-MDT0001 complete [ 6615.483051] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 6617.687293] Key type lgssc unregistered [ 6617.954173] LNet: 260145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6617.963639] LNetError: 260145:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6617.986701] LNet: Removed LNI 192.168.201.159@tcp [ 6618.672568] Key type .llcrypt unregistered [ 6618.674941] Key type ._llcrypt unregistered [ 6652.001091] Key type ._llcrypt registered [ 6652.002580] Key type .llcrypt registered [ 6652.108987] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 6664.120906] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 6664.921397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6664.933666] alg: No test for adler32 (adler32-zlib) [ 6665.992369] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 6666.202821] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 6667.865130] Key type lgssc registered [ 6668.832395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6677.160967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6683.178176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6688.382775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6694.044989] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6700.383352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6700.448910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6700.459726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6701.677642] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6701.711816] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6701.799075] Lustre: lustre-MDT0000: new disk, initializing [ 6701.870731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6701.889223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6705.065363] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6713.866398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6713.935731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6714.058151] Lustre: 264058:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6714.124778] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6714.128487] Lustre: Skipped 1 previous similar message [ 6714.234766] Lustre: lustre-MDT0001: new disk, initializing [ 6714.351558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6714.418947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6714.427813] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6717.510917] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6723.481848] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6726.522640] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6732.665741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6732.719776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6732.898792] Lustre: lustre-OST0000: new disk, initializing [ 6732.903147] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6732.984576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6738.204372] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6738.523475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6738.538724] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6738.650877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6745.890832] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6753.014136] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 6763.748217] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 6773.188500] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 6781.826292] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 6825.955745] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6825.960073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6825.967211] LustreError: Skipped 1 previous similar message [ 6825.968973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6825.981561] Lustre: Skipped 1 previous similar message [ 6831.073323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6831.078303] Lustre: Skipped 2 previous similar messages [ 6836.199135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6836.201738] Lustre: Skipped 1 previous similar message [ 6839.265365] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6839.316137] LustreError: 267678:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6839.542470] Lustre: server umount lustre-OST0000 complete [ 6846.435872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6846.443916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6846.450468] Lustre: Skipped 1 previous similar message [ 6846.460076] Lustre: Skipped 1 previous similar message [ 6851.557394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6851.559908] Lustre: Skipped 1 previous similar message [ 6857.184125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6857.297661] LustreError: 267880:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6857.300236] LustreError: 267880:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6857.373266] Lustre: server umount lustre-MDT0000 complete [ 6860.725226] LustreError: 264048:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347040 with bad export cookie 12484079455411646056 [ 6860.732731] LustreError: 264048:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6860.733161] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6860.936776] LustreError: 268083:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6860.948655] LustreError: 268083:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6861.139795] Lustre: server umount lustre-MDT0001 complete [ 6869.833820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6874.014363] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6878.990698] LustreError: 269026:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6878.993756] LustreError: 269026:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6879.088372] Lustre: server umount MGS complete [ 6889.112726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6889.673383] LustreError: 269663:0:(qmt_pool.c:1523:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 6889.772754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6893.748976] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6895.074211] LustreError: 269651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6895.090169] LustreError: 269651:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6900.197181] LustreError: 269651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6901.728949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6902.086353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6905.634806] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6912.135809] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6916.132811] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6923.647775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6924.023864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6929.290720] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6934.697886] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6966.242655] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6966.245686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6966.264949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6966.267455] Lustre: Skipped 2 previous similar messages [ 6971.001470] LustreError: 272310:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6971.003557] LustreError: 272310:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6971.094372] Lustre: server umount lustre-OST0000 complete [ 6976.484125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6976.491669] Lustre: Skipped 2 previous similar messages [ 6980.389437] LustreError: 272512:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6980.391904] LustreError: 272512:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6980.496636] Lustre: server umount lustre-MDT0000 complete [ 6981.605138] LustreError: 269646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6981.635891] LustreError: 269646:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6983.684082] LustreError: 269630:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347163 with bad export cookie 12484079455411647995 [ 6983.692839] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6983.701261] LustreError: 269630:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6984.084369] Lustre: server umount lustre-MDT0001 complete [ 6994.354222] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 6997.402321] Key type lgssc unregistered [ 6997.668821] LNet: 273244:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6997.690363] LNetError: 273244:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6998.694524] LNet: Removed LNI 192.168.201.159@tcp [ 6999.580253] Key type .llcrypt unregistered [ 6999.587801] Key type ._llcrypt unregistered [ 7020.426232] Key type ._llcrypt registered [ 7020.427503] Key type .llcrypt registered [ 7020.564613] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 7033.347942] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 7034.257978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7034.266366] alg: No test for adler32 (adler32-zlib) [ 7035.384606] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 7035.706746] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 7037.424220] Key type lgssc registered [ 7038.841439] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7049.201734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7056.532738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7068.112763] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 7077.668132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7084.326406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7084.380280] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7084.388693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7085.605232] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7085.641408] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7085.685623] Lustre: lustre-MDT0000: new disk, initializing [ 7085.775907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7085.788443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7089.231539] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7098.702964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7098.772912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7098.990906] Lustre: lustre-OST0000: new disk, initializing [ 7098.997982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7099.001894] Lustre: Skipped 1 previous similar message [ 7099.086202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7101.026472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7101.034080] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7101.066428] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7104.131774] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8012.263169] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8012.276269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8016.508505] LustreError: 278273:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 8016.726518] Lustre: server umount lustre-MDT0000 complete [ 8020.300499] LustreError: 276416:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348199 with bad export cookie 10067928318786642734 [ 8020.309450] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8020.351621] LustreError: 276416:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8020.392613] LustreError: 278477:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8020.398979] LustreError: 278477:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8020.538041] Lustre: server umount lustre-OST0000 complete [ 8043.707734] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 8052.817617] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 8062.835432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8070.040873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8077.413174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8083.325395] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8088.659141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8088.720555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8088.942596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8088.970234] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8089.023509] Lustre: lustre-MDT0000: new disk, initializing [ 8089.103656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8089.122212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8092.981805] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8104.301735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8104.377486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8104.601440] Lustre: 281522:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8104.658523] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8104.660491] Lustre: Skipped 1 previous similar message [ 8104.715478] Lustre: lustre-MDT0001: new disk, initializing [ 8104.834469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8104.895177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8104.907936] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8108.947451] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8116.920990] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8120.224355] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8125.738890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8125.792698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8126.002684] Lustre: lustre-OST0000: new disk, initializing [ 8126.008000] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8126.099996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8131.655689] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8133.169781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8133.174939] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8133.281736] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8139.544794] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8143.430194] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8143.653493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8147.712054] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8147.952154] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8153.569548] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8153.572729] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8153.585343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8154.082967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8155.648543] LustreError: 284464:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8155.655705] LustreError: 284464:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8155.905286] Lustre: server umount lustre-OST0000 complete [ 8161.262769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8161.274883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8161.311379] Lustre: Skipped 1 previous similar message [ 8161.323813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8161.334466] Lustre: Skipped 2 previous similar messages [ 8164.321033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8164.329426] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8166.284753] LustreError: 284665:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8166.287147] LustreError: 284665:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8166.404740] Lustre: server umount lustre-MDT0000 complete [ 8170.559576] LustreError: 281516:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348349 with bad export cookie 10067928318789583581 [ 8170.568908] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8170.586993] LustreError: 281516:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8170.845112] LustreError: 284867:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8170.883881] LustreError: 284867:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8171.301889] Lustre: server umount lustre-MDT0001 complete [ 8175.601042] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [ 8177.446879] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 21:59:14 (1763348354) [ 8183.892808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8184.499951] LustreError: 285441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8184.545920] LustreError: 285441:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8184.600665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8188.027729] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8189.925816] LustreError: 285441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8194.021483] LustreError: 285441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8195.743223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8196.067880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8199.388378] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8205.549211] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8209.583215] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8217.439909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8217.848667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8224.137296] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8229.058422] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8234.880097] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8234.942706] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8235.057989] Lustre: lustre-OST0001: new disk, initializing [ 8235.061421] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8235.122921] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8240.670640] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8244.305266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8244.316671] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8244.370638] LustreError: 288463:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 8245.412170] LustreError: 288463:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 8245.430933] LustreError: 288463:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 8246.441610] LustreError: 288457:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 8248.482043] LustreError: 288457:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 8248.505135] LustreError: 288457:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 8250.228382] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8250.389096] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8252.648133] LustreError: 288463:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 8252.652745] LustreError: 288463:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 8 previous similar messages [ 8253.901769] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff93dd0679e000.ost_server_uuid 50 [ 8255.537822] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff93dd0679e000.ost_server_uuid in FULL state after 0 sec [ 8259.089576] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93dd0679e000.ost_server_uuid 50 [ 8260.688521] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff93dd0679e000.ost_server_uuid in FULL state after 0 sec [ 8260.853435] LustreError: 288463:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 8260.866148] LustreError: 288463:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [ 8261.941052] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8275.432058] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8275.437259] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8275.449582] Lustre: Skipped 1 previous similar message [ 8275.467479] Lustre: Skipped 2 previous similar messages [ 8285.665695] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8285.680455] Lustre: Skipped 5 previous similar messages [ 8289.249546] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8289.297322] LustreError: 289251:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 8289.299297] LustreError: 289251:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8289.366819] Lustre: server umount lustre-OST0001 complete [ 8290.788269] LustreError: 287037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8290.809787] LustreError: 287037:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8295.909792] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8295.917097] LustreError: 287037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8295.918714] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8295.924908] Lustre: Skipped 1 previous similar message [ 8295.945820] LustreError: 287037:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8300.764966] Lustre: server umount lustre-OST0000 complete [ 8306.157080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8306.171368] Lustre: Skipped 4 previous similar messages [ 8310.526479] LustreError: 289666:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8310.537851] LustreError: 289666:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8310.661179] Lustre: server umount lustre-MDT0000 complete [ 8311.266511] LustreError: 285437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8311.292366] LustreError: 285437:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8314.484079] LustreError: 285421:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348493 with bad export cookie 10067928318789584463 [ 8314.496127] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8314.508281] LustreError: 285421:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8314.985304] Lustre: server umount lustre-MDT0001 complete [ 8323.813747] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 8326.334903] Key type lgssc unregistered [ 8326.649529] LNet: 290399:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8326.660736] LNetError: 290399:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8326.678289] LNet: Removed LNI 192.168.201.159@tcp [ 8327.522714] Key type .llcrypt unregistered [ 8327.526945] Key type ._llcrypt unregistered [ 8352.302751] Key type ._llcrypt registered [ 8352.311928] Key type .llcrypt registered [ 8352.407581] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 8353.401480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8353.533671] alg: No test for adler32 (adler32-zlib) [ 8354.874668] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 8355.156823] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 8356.856178] Key type lgssc registered [ 8358.053191] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8365.542846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8365.560895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8367.001505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8370.789852] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8376.560767] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8377.321586] LustreError: 291804:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8377.335797] LustreError: 291804:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8382.436048] LustreError: 291804:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8384.138944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8384.675019] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8388.437189] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8397.602536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8397.871377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8403.210869] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8408.448663] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8409.132558] LustreError: 293356:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8412.210613] LustreError: 293359:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8412.218107] LustreError: 293359:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8412.223884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 8414.763353] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8415.052905] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8420.353808] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 8420.408322] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8425.292190] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8430.241069] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 8430.424244] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8538.084175] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8538.086925] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8538.124591] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8538.134470] Lustre: Skipped 1 previous similar message [ 8543.208919] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8543.211777] Lustre: Skipped 1 previous similar message [ 8548.320491] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8548.327253] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8548.337968] Lustre: Skipped 1 previous similar message [ 8548.369773] LustreError: 295232:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 8548.423561] Lustre: server umount lustre-OST0001 complete [ 8553.444983] LustreError: 293358:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8553.445547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8553.446977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8553.453955] LustreError: 293358:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8553.476478] Lustre: Skipped 2 previous similar messages [ 8558.563482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8558.566291] Lustre: Skipped 2 previous similar messages [ 8563.690391] LustreError: 293358:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8563.703781] LustreError: 293358:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8567.264181] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8567.297196] LustreError: 295435:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 8567.302932] LustreError: 295435:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8567.402636] Lustre: server umount lustre-OST0000 complete [ 8573.922700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8573.925877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8573.926067] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8573.926072] LustreError: Skipped 1 previous similar message [ 8573.928111] Lustre: Skipped 1 previous similar message [ 8573.935786] Lustre: Skipped 2 previous similar messages [ 8576.173627] LustreError: 295637:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 8576.178503] LustreError: 295637:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8576.249531] Lustre: server umount lustre-MDT0000 complete [ 8579.571065] LustreError: 295129:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348758 with bad export cookie 5574725034532550037 [ 8579.574747] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8579.581467] LustreError: 295129:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8579.810699] LustreError: 295839:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 8579.821254] LustreError: 295839:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8580.000524] Lustre: server umount lustre-MDT0001 complete [ 8589.366341] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 8591.801552] Key type lgssc unregistered [ 8592.117105] LNet: 296371:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8592.132264] LNetError: 296371:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8593.191373] LNet: Removed LNI 192.168.201.159@tcp [ 8593.857245] Key type .llcrypt unregistered [ 8593.858439] Key type ._llcrypt unregistered [ 8631.368980] Key type ._llcrypt registered [ 8631.375756] Key type .llcrypt registered [ 8631.503777] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 8632.498167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8632.736217] alg: No test for adler32 (adler32-zlib) [ 8633.817537] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 8634.086963] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 8635.768178] Key type lgssc registered [ 8637.008906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8645.508721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8645.521649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8646.979501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8650.032725] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8657.384450] LustreError: 298639:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8657.425646] LustreError: 298639:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8658.662346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8659.078071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8663.711675] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8666.554248] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8672.476986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8672.803975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8677.834871] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8685.108864] LustreError: 300101:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8685.210612] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8685.386420] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8690.248679] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8691.693656] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 8691.720326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 8698.424820] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8704.826984] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8722.405631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8722.412783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8722.425672] Lustre: Skipped 2 previous similar messages [ 8722.437410] Lustre: Skipped 3 previous similar messages [ 8727.169246] LustreError: 302049:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 8727.282802] Lustre: server umount lustre-MDT0000 complete [ 8727.533230] LustreError: 302050:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8727.551288] LustreError: 302050:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8730.399024] LustreError: 298619:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348909 with bad export cookie 4638736381053074818 [ 8730.404149] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8730.422184] LustreError: 298619:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8730.573823] LustreError: 302251:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 8730.580150] LustreError: 302251:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8730.687588] Lustre: server umount lustre-MDT0001 complete [ 8743.737467] LustreError: 302451:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 8743.740962] LustreError: 302451:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8743.813701] Lustre: server umount lustre-OST0000 complete [ 8757.725879] LustreError: 302653:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8757.734822] LustreError: 302653:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8757.908046] Lustre: server umount lustre-OST0001 complete [ 8775.540321] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 8785.221912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8785.515253] LustreError: 303832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8785.574250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8788.373327] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8791.010652] LustreError: 303832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8795.358952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8795.752922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8799.019520] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8801.647844] Lustre: 304940:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8807.641479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8810.153260] LustreError: 305297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8810.160722] LustreError: 305297:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8810.170902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 8813.970816] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8822.698359] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8823.089206] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8823.095704] Lustre: Skipped 1 previous similar message [ 8825.049829] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 8828.218213] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8834.256110] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8836.891992] Lustre: 306788:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8846.818658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8846.821967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8846.845639] Lustre: Skipped 1 previous similar message [ 8846.849867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8848.875035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8848.876176] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8848.893205] Lustre: Skipped 2 previous similar messages [ 8848.899067] Lustre: Skipped 2 previous similar messages [ 8851.206127] LustreError: 306894:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 8851.209196] LustreError: 306894:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8851.334502] Lustre: server umount lustre-MDT0000 complete [ 8853.997353] LustreError: 306807:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8854.009109] LustreError: 306807:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8854.673416] LustreError: 303811:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349034 with bad export cookie 4638736381053076519 [ 8854.675674] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8854.682996] LustreError: 303811:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8855.001909] Lustre: server umount lustre-MDT0001 complete [ 8868.363476] LustreError: 307297:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 8868.367302] LustreError: 307297:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 8868.403830] Lustre: server umount lustre-OST0000 complete [ 8882.117862] Lustre: server umount lustre-OST0001 complete [ 8885.390618] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [ 8886.605330] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [ 8887.838037] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 22:11:05 (1763349065) [ 8904.468251] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 8910.375105] loop0: detected capacity change from 0 to 1099511627776 [ 8910.662134] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 8910.666787] *** VALIDATE xfs *** [ 8910.701562] XFS (loop0): Mounting V5 Filesystem [ 8910.719595] XFS (loop0): Ending clean mount [ 8911.751403] loop1: detected capacity change from 0 to 18691697672192 [ 8958.823262] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8975.760531] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 8985.765477] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 8993.501528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8998.672145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9003.964296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9009.026681] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9015.063886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9015.142976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9015.464300] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9015.513334] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9015.575045] Lustre: lustre-MDT0000: new disk, initializing [ 9015.708796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9015.754435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9020.325601] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9029.421510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9029.496086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9029.655231] Lustre: 311883:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9029.730076] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9029.747642] Lustre: Skipped 1 previous similar message [ 9029.890362] Lustre: lustre-MDT0001: new disk, initializing [ 9030.035655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9030.058837] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9034.108499] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9041.580547] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9045.733268] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9051.746682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9051.817366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9052.126868] Lustre: lustre-OST0000: new disk, initializing [ 9052.133111] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9054.190639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9054.204784] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9054.371189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9057.823460] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9065.391622] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9070.296590] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9070.527805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9074.092473] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9074.381384] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9079.780528] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9079.783819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9079.786765] LustreError: Skipped 1 previous similar message [ 9079.794628] Lustre: Skipped 1 previous similar message [ 9079.797108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9079.809266] Lustre: Skipped 2 previous similar messages [ 9082.117364] LustreError: 314820:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9082.134797] LustreError: 314820:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 9082.301994] Lustre: server umount lustre-OST0000 complete [ 9086.433176] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9086.439917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9086.466047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9091.750387] Lustre: server umount lustre-MDT0000 complete [ 9095.080352] LustreError: 313627:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349274 with bad export cookie 4638736381053078073 [ 9095.095381] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9095.100960] LustreError: 313627:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9095.477725] Lustre: server umount lustre-MDT0001 complete [ 9102.827528] XFS (loop0): Unmounting Filesystem [ 9106.725483] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 22:14:44 (1763349284) [ 9111.918185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9112.306257] LustreError: 316039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9112.322275] LustreError: 316039:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9112.379138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9112.382240] Lustre: Skipped 2 previous similar messages [ 9115.691897] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9123.307591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9126.792201] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9131.401638] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9134.811282] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9140.908136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9146.290558] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9150.598686] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9168.869094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9168.876680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9168.883670] Lustre: Skipped 1 previous similar message [ 9168.888195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9168.891240] Lustre: Skipped 2 previous similar messages [ 9172.963435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9174.679253] LustreError: 318404:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9174.681842] LustreError: 318404:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 9174.772941] Lustre: server umount lustre-OST0000 complete [ 9184.217900] Lustre: server umount lustre-MDT0000 complete [ 9187.256903] LustreError: 316019:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349366 with bad export cookie 4638736381053078955 [ 9187.263089] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9187.579975] Lustre: server umount lustre-MDT0001 complete [ 9195.467660] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 9197.898316] Key type lgssc unregistered [ 9198.144187] LNet: 319338:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9198.152535] LNetError: 319338:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9198.176390] LNet: Removed LNI 192.168.201.159@tcp [ 9199.038393] Key type .llcrypt unregistered [ 9199.044688] Key type ._llcrypt unregistered [ 9225.599109] Key type ._llcrypt registered [ 9225.601051] Key type .llcrypt registered [ 9225.691110] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 9226.489758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9226.576726] alg: No test for adler32 (adler32-zlib) [ 9227.595947] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 9227.787257] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 9229.458971] Key type lgssc registered [ 9230.282877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9236.504315] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9236.523767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9237.963723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9242.288157] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9248.233598] LustreError: 320867:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9248.240875] LustreError: 320867:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9252.860702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9253.451687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9257.867559] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9263.071457] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9266.554800] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9274.572229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9274.986269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9281.236406] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9286.300955] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9298.920972] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9298.928841] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9298.931529] Lustre: Skipped 1 previous similar message [ 9300.567440] LustreError: 323161:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9300.626823] Lustre: server umount lustre-MDT0001 complete [ 9303.351923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9304.037149] LustreError: 320862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9304.065642] LustreError: 320862:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9309.160567] LustreError: 320863:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9309.176131] LustreError: 320863:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9311.245503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9311.402723] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9311.406376] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9311.463644] Lustre: 320854:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9311.901710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9315.332191] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9317.376710] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9320.958179] LustreError: 324141:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 9320.962081] LustreError: 324141:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 9320.976028] LustreError: 320858:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 9321.032015] LustreError: 324144:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.201.159@tcp: cfg command failed: rc = -17 [ 9321.048733] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 9321.048733] [ 9621.667979] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.59@tcp (stopping) [ 9621.673524] Lustre: Skipped 1 previous similar message [ 9624.551314] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9624.557825] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9624.559769] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9624.570055] Lustre: Skipped 2 previous similar messages [ 9626.458262] LustreError: 332531:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9626.468232] LustreError: 332531:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9626.592685] Lustre: server umount lustre-MDT0001 complete [ 9626.776088] LustreError: 322918:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9626.792425] LustreError: 322918:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9629.148823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9631.896427] LustreError: 320863:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9631.919990] LustreError: 320863:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9636.063024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9636.148294] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9636.158549] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9636.197995] Lustre: 320855:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9636.448282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9639.293486] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9641.456228] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9644.546490] LustreError: 333561:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 9644.552376] LustreError: 333561:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 9644.559463] LustreError: 320858:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 9644.563544] LustreError: 320858:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 9644.602116] LustreError: 333564:0:(obd_config.c:689:class_setup()) Device 8 already setup (type osp) [ 9644.605424] LustreError: 333564:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.201.159@tcp: cfg command failed: rc = -17 [ 9644.617379] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.201.159@tcp [ 9644.617379] [ 9943.525163] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9943.526750] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9943.540026] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9943.540035] Lustre: Skipped 1 previous similar message [ 9948.643878] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9948.649775] Lustre: Skipped 3 previous similar messages [ 9948.651175] LustreError: 341799:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9948.654951] LustreError: 341799:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9948.716743] Lustre: server umount lustre-MDT0001 complete [ 9949.336151] LustreError: 329227:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9949.348150] LustreError: 329227:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9951.137411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9957.199395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9957.295586] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9957.298101] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9957.323905] Lustre: 320856:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9957.555254] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 9960.887637] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9962.986040] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9964.026912] LustreError: 342766:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 9964.032433] LustreError: 342766:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 9964.060626] LustreError: 320858:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 9964.070277] LustreError: 320858:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 9964.143585] LustreError: 342769:0:(lod_lov.c:207:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 9964.160758] LustreError: 342769:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.201.159@tcp: cfg command failed: rc = -114 [ 9964.170736] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 9964.170736] [10285.540991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10285.543693] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10285.553180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10285.564386] Lustre: Skipped 2 previous similar messages [10287.363458] LustreError: 351234:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10287.367281] LustreError: 351234:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10287.454562] Lustre: server umount lustre-MDT0000 complete [10290.382440] LustreError: 320847:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350469 with bad export cookie 12307855458297105426 [10290.383269] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10290.387497] LustreError: 320847:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10290.667419] LustreError: 329227:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10290.668456] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10290.694138] LustreError: 329227:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10290.700266] Lustre: Skipped 1 previous similar message [10296.594814] LustreError: 351436:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10296.602428] LustreError: 351436:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10296.786663] Lustre: server umount lustre-MDT0001 complete [10300.389609] Lustre: server umount lustre-OST0000 complete [10308.086765] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [10315.815491] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [10323.370638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10329.769444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10335.651602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10341.315543] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10347.113713] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 22:35:24 (1763350524) [10352.746851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10352.806558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10352.995565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10353.029962] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10353.101558] Lustre: lustre-MDT0000: new disk, initializing [10353.180826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10353.195278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10355.946056] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10364.653938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10364.710280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10364.798314] Lustre: 354175:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10364.851446] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10364.855435] Lustre: Skipped 1 previous similar message [10364.928531] Lustre: lustre-MDT0001: new disk, initializing [10364.997351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10365.034151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10365.040334] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10367.815400] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10374.083929] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10377.526154] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10383.471231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10383.544888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10383.853310] Lustre: lustre-OST0000: new disk, initializing [10383.856949] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10383.954183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10389.108510] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10391.107162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10391.121734] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10391.199621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10395.856805] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10401.764748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10401.774635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10401.783249] Lustre: Skipped 1 previous similar message [10401.791438] Lustre: Skipped 6 previous similar messages [10407.759708] LustreError: 356620:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10407.763922] LustreError: 356620:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [10407.894232] Lustre: server umount lustre-MDT0000 complete [10411.032760] LustreError: 354968:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [10411.033173] LustreError: 354167:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350590 with bad export cookie 12307855458297107386 [10411.038856] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10411.074087] LustreError: 354167:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10411.346813] Lustre: server umount lustre-MDT0001 complete [10423.821754] LustreError: 357023:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [10423.832516] LustreError: 357023:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [10424.060871] Lustre: server umount lustre-OST0000 complete [10453.867181] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [10463.944310] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [10476.188287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10482.782336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10488.879108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10494.709705] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10496.472739] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10286 sec ========= 22:37:53 (1763350673) [10497.858215] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:37:55 (1763350675) === [10500.862468] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:37:58 (1763350678) === [10535.390205] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [10537.993308] Key type lgssc unregistered [10538.283993] LNet: 362136:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10538.300540] LNetError: 362136:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10538.317959] LNet: Removed LNI 192.168.201.159@tcp [10539.156776] Key type .llcrypt unregistered [10539.158355] Key type ._llcrypt unregistered