[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 731373670 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2656MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002382] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008017] pid_max: default: 32768 minimum: 301 [ 0.010161] LSM: Security Framework initializing [ 0.011062] Yama: becoming mindful. [ 0.012037] SELinux: Initializing. [ 0.013096] *** VALIDATE selinux *** [ 0.022732] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028563] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029173] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030123] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031105] *** VALIDATE tmpfs *** [ 0.032000] *** VALIDATE proc *** [ 0.032000] *** VALIDATE cgroup *** [ 0.032012] *** VALIDATE cgroup2 *** [ 0.033289] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034186] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037016] Spectre V2 : User space: Vulnerable [ 0.038009] Speculative Store Bypass: Vulnerable [ 0.040884] debug: unmapping init [mem 0xffffffffbe259000-0xffffffffbe260fff] [ 0.044105] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045790] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046033] ... version: 2 [ 0.047015] ... bit width: 48 [ 0.048015] ... generic registers: 4 [ 0.049010] ... value mask: 0000ffffffffffff [ 0.050025] ... max period: 00007fffffffffff [ 0.051026] ... fixed-purpose events: 3 [ 0.052017] ... event mask: 000000070000000f [ 0.053302] rcu: Hierarchical SRCU implementation. [ 0.055508] smp: Bringing up secondary CPUs ... [ 0.056698] x86: Booting SMP configuration: [ 0.057031] .... node #0, CPUs: #1 #2 #3 [ 0.065041] smp: Brought up 1 node, 4 CPUs [ 0.067029] smpboot: Max logical packages: 1 [ 0.068017] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.111615] node 0 deferred pages initialised in 38ms [ 0.117014] devtmpfs: initialized [ 0.118371] x86/mm: Memory block size: 128MB [ 0.121909] gcov: version magic: 0x41383552 [ 0.125928] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.129116] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.133324] pinctrl core: initialized pinctrl subsystem [ 0.136242] [ 0.137021] ************************************************************* [ 0.139014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.142016] ** ** [ 0.144009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.146013] ** ** [ 0.149010] ** This means that this kernel is built to expose internal ** [ 0.152013] ** IOMMU data structures, which may compromise security on ** [ 0.154017] ** your system. ** [ 0.157019] ** ** [ 0.160017] ** If you see this message and you are not debugging the ** [ 0.163017] ** kernel, report this immediately to your vendor! ** [ 0.166015] ** ** [ 0.168014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171031] ************************************************************* [ 0.177059] NET: Registered protocol family 16 [ 0.180466] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.182068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.184059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.188279] cpuidle: using governor menu [ 0.191263] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.193510] PCI: Using configuration type 1 for base access [ 0.196125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.206171] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.208020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.213179] cryptd: max_cpu_qlen set to 1000 [ 0.219252] ACPI: Added _OSI(Module Device) [ 0.221031] ACPI: Added _OSI(Processor Device) [ 0.223015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.226016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.232053] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.239881] ACPI: Interpreter enabled [ 0.240000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.242012] ACPI: Using IOAPIC for interrupt routing [ 0.244101] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.249445] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.260850] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.263049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.266020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.268093] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.274490] acpiphp: Slot [2] registered [ 0.276151] acpiphp: Slot [3] registered [ 0.278091] acpiphp: Slot [4] registered [ 0.279100] acpiphp: Slot [5] registered [ 0.281204] acpiphp: Slot [6] registered [ 0.283120] acpiphp: Slot [7] registered [ 0.285136] acpiphp: Slot [8] registered [ 0.288791] acpiphp: Slot [9] registered [ 0.290182] acpiphp: Slot [10] registered [ 0.291000] acpiphp: Slot [11] registered [ 0.292018] acpiphp: Slot [12] registered [ 0.294106] acpiphp: Slot [13] registered [ 0.296097] acpiphp: Slot [14] registered [ 0.297092] acpiphp: Slot [15] registered [ 0.299100] acpiphp: Slot [16] registered [ 0.301112] acpiphp: Slot [17] registered [ 0.302084] acpiphp: Slot [18] registered [ 0.304094] acpiphp: Slot [19] registered [ 0.306088] acpiphp: Slot [20] registered [ 0.307134] acpiphp: Slot [21] registered [ 0.309103] acpiphp: Slot [22] registered [ 0.310444] acpiphp: Slot [23] registered [ 0.313138] acpiphp: Slot [24] registered [ 0.314090] acpiphp: Slot [25] registered [ 0.317125] acpiphp: Slot [26] registered [ 0.319127] acpiphp: Slot [27] registered [ 0.321088] acpiphp: Slot [28] registered [ 0.323137] acpiphp: Slot [29] registered [ 0.324135] acpiphp: Slot [30] registered [ 0.326093] acpiphp: Slot [31] registered [ 0.328093] PCI host bridge to bus 0000:00 [ 0.329060] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.332021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.335021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.338021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.340024] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.354027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.358175] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.366280] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.368538] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.375000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.383057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.384017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.386016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.390017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.392424] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.394692] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.398053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.401653] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.406000] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.417016] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.422771] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.429591] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.439022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.446017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.465019] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.475766] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.481015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.487015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.507019] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.523428] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.530017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.539026] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.559017] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.567000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.571016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.575015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.578022] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.581000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.584022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.588015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.598015] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.606000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.611015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.620026] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.631017] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.639000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.644366] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.646381] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.649407] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.651257] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.657032] iommu: Default domain type: Passthrough [ 0.658435] SCSI subsystem initialized [ 0.659186] ACPI: bus type USB registered [ 0.661106] usbcore: registered new interface driver usbfs [ 0.663110] usbcore: registered new interface driver hub [ 0.666094] usbcore: registered new device driver usb [ 0.667252] pps_core: LinuxPPS API ver. 1 registered [ 0.669012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.672073] PTP clock support registered [ 0.678031] EDAC MC: Ver: 3.0.0 [ 0.681124] PCI: Using ACPI for IRQ routing [ 0.685162] NetLabel: Initializing [ 0.687012] NetLabel: domain hash size = 128 [ 0.689025] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.691085] NetLabel: unlabeled traffic allowed by default [ 0.694177] vgaarb: loaded [ 0.700349] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.701010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.707461] clocksource: Switched to clocksource kvm-clock [ 1.011297] VFS: Disk quotas dquot_6.6.0 [ 1.013376] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.017950] *** VALIDATE ramfs *** [ 1.021436] *** VALIDATE hugetlbfs *** [ 1.023924] pnp: PnP ACPI init [ 1.028391] pnp: PnP ACPI: found 6 devices [ 1.058510] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.064563] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.068665] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.071773] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.074882] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.077744] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 1.080847] NET: Registered protocol family 2 [ 1.083994] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.088711] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.094279] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.100205] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.104140] TCP: Hash tables configured (established 65536 bind 65536) [ 1.107129] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.110777] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.113912] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.117857] NET: Registered protocol family 1 [ 1.121687] RPC: Registered named UNIX socket transport module. [ 1.124618] RPC: Registered udp transport module. [ 1.126699] RPC: Registered tcp transport module. [ 1.128842] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.132031] NET: Registered protocol family 44 [ 1.133695] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.136255] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.139365] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.142665] PCI: CLS 0 bytes, default 64 [ 1.144979] Unpacking initramfs... [ 3.521859] debug: unmapping init [mem 0xffffa0d9bcc54000-0xffffa0d9bffbffff] [ 3.526827] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.528794] software IO TLB: mapped [mem 0x00000000a2000000-0x00000000a6000000] (64MB) [ 3.532509] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 4.480559] Initialise system trusted keyrings [ 4.486123] Key type blacklist registered [ 4.488307] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.501421] zbud: loaded [ 4.504659] *** VALIDATE nfs *** [ 4.505514] *** VALIDATE nfs4 *** [ 4.506867] pstore: using deflate compression [ 4.510480] Platform Keyring initialized [ 4.760097] NET: Registered protocol family 38 [ 4.763700] Key type asymmetric registered [ 4.767771] Asymmetric key parser 'x509' registered [ 4.780604] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.805445] io scheduler mq-deadline registered [ 4.806865] io scheduler kyber registered [ 4.808200] io scheduler bfq registered [ 4.813207] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.817275] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.824811] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.830822] ACPI: Power Button [PWRF] [ 5.004314] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.128631] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.467948] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.691362] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.025399] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.070131] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.129806] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.138337] Non-volatile memory driver v1.3 [ 6.139933] Linux agpgart interface v0.103 [ 6.193888] virtio_blk virtio1: [vda] 133248 512-byte logical blocks (68.2 MB/65.1 MiB) [ 6.197233] vda: detected capacity change from 0 to 68222976 [ 6.230193] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.235650] vdb: detected capacity change from 0 to 1073741824 [ 6.262569] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.267753] vdc: detected capacity change from 0 to 2621440000 [ 6.292624] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.297755] vdd: detected capacity change from 0 to 2621440000 [ 6.331209] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.338534] vde: detected capacity change from 0 to 4294967296 [ 6.364959] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.370270] vdf: detected capacity change from 0 to 4294967296 [ 6.385626] libphy: Fixed MDIO Bus: probed [ 6.395268] usbcore: registered new interface driver usbserial_generic [ 6.400081] usbserial: USB Serial support registered for generic [ 6.402371] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.417548] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.422472] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.427967] mousedev: PS/2 mouse device common for all mice [ 6.436185] rtc_cmos 00:05: RTC can wake from S4 [ 6.438543] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.451094] rtc_cmos 00:05: registered as rtc0 [ 6.455376] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.464949] intel_pstate: CPU model not supported [ 6.478729] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.488236] hid: raw HID events driver (C) Jiri Kosina [ 6.494143] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.501959] usbcore: registered new interface driver usbhid [ 6.506639] usbhid: USB HID core driver [ 6.509428] drop_monitor: Initializing network drop monitor service [ 6.514962] Initializing XFRM netlink socket [ 6.518319] NET: Registered protocol family 10 [ 6.525460] Segment Routing with IPv6 [ 6.527019] NET: Registered protocol family 17 [ 6.529834] mpls_gso: MPLS GSO support [ 6.538298] RAS: Correctable Errors collector initialized. [ 6.542355] AVX version of gcm_enc/dec engaged. [ 6.545509] AES CTR mode by8 optimization enabled [ 6.657928] sched_clock: Marking stable (6657695324, 0)->(8290183410, -1632488086) [ 6.666923] registered taskstats version 1 [ 6.670147] Loading compiled-in X.509 certificates [ 6.673298] zswap: loaded using pool lzo/zbud [ 6.719171] Key type big_key registered [ 6.749526] Key type encrypted registered [ 6.752227] ima: No TPM chip found, activating TPM-bypass! [ 6.755195] ima: Allocated hash algorithm: sha1 [ 6.756882] ima: No architecture policies found [ 6.762275] evm: Initialising EVM extended attributes: [ 6.764403] evm: security.selinux [ 6.765753] evm: security.ima [ 6.766949] evm: security.capability [ 6.768371] evm: HMAC attrs: 0x1 [ 6.772670] rtc_cmos 00:05: setting system clock to 2025-10-10 19:42:35 UTC (1760125355) [ 6.779066] debug: unmapping init [mem 0xffffffffbf203000-0xffffffffbf3fffff] [ 6.782222] debug: unmapping init [mem 0xffffffffbdf82000-0xffffffffbe258fff] [ 6.791070] Write protecting the kernel read-only data: 28672k [ 6.797071] debug: unmapping init [mem 0xffffffffbc603000-0xffffffffbc7fffff] [ 6.806904] debug: unmapping init [mem 0xffffffffbcf14000-0xffffffffbcffffff] [ 6.853431] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.861897] systemd[1]: Detected virtualization kvm. [ 6.864983] systemd[1]: Detected architecture x86-64. [ 6.866894] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.901392] systemd[1]: No hostname configured. [ 6.903308] systemd[1]: Set hostname to . [ 6.909621] random: systemd: uninitialized urandom read (16 bytes read) [ 6.914166] systemd[1]: Initializing machine ID from random generator. [ 6.975312] random: ln: uninitialized urandom read (6 bytes read) [ 7.267835] random: systemd: uninitialized urandom read (16 bytes read) [ 7.277883] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 7.293042] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 7.311974] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.014463] device-mapper: uevent: version 1.0.3 [ 9.024931] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 10.700209] virtio_net virtio0 ens2: renamed from eth0 [ 10.928803] random: fast init done [ 11.008254] scsi host0: ata_piix [ 11.108175] scsi host1: ata_piix [ 11.138817] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.144537] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 17.292198] random: crng init done [ 17.294275] random: 7 urandom warning(s) missed due to ratelimiting [ 19.718709] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 21.911265] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.906751] printk: systemd: 26 output lines suppressed due to ratelimiting [ 25.474857] SELinux: Disabled at runtime. [ 25.570207] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 25.585729] systemd[1]: Detected virtualization kvm. [ 25.590701] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 27.273346] systemd[1]: initrd-switch-root.service: Succeeded. [ 27.281905] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 27.298492] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 27.312679] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 27.323324] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 27.350277] systemd[1]: Starting Journal Service... Starting Journal Service... [ 27.398317] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ 28.546612] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 30.379508] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 32.045875] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 32.100497] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 33.724575] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 33.787455] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit)[ 39.147986] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (12s / no limit)[ 39.913686] NFS: Registering the id_resolver key type [ 39.920887] Key type id_resolver registered [ 39.925352] Key type id_legacy registered [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg412-server login: [ 83.320533] libcfs: loading out-of-tree module taints kernel. [ 83.349072] Key type ._llcrypt registered [ 83.353544] Key type .llcrypt registered [ 83.437801] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 94.583916] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 95.288064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 95.295043] alg: No test for adler32 (adler32-zlib) [ 96.388479] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 96.776507] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 98.457095] Key type lgssc registered [ 99.292425] Lustre: Echo OBD driver; http://www.lustre.org/ [ 108.667147] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 110.109664] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 117.574723] hrtimer: interrupt took 7712109 ns [ 117.881719] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 125.028336] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 133.056858] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 144.893527] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 154.030870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 154.081633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 154.125924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 155.315315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 155.336430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 155.419917] Lustre: lustre-MDT0000: new disk, initializing [ 155.495516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 155.508776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 157.959734] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 167.154529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 167.200292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 167.244811] Lustre: 6488:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 167.263513] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 167.266914] Lustre: Skipped 1 previous similar message [ 167.317343] Lustre: lustre-MDT0001: new disk, initializing [ 167.359665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 167.375411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 167.383432] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 169.537881] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 172.962547] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 179.599806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 179.648810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 179.787169] Lustre: lustre-OST0000: new disk, initializing [ 179.793459] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 179.829919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 183.228349] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 183.370227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 183.383417] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 183.457424] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 192.341139] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 192.400708] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 192.468353] Lustre: lustre-OST0001: new disk, initializing [ 192.470508] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 192.524934] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 193.233830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 193.239932] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 193.277600] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 196.186629] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.670785] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 209.351246] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 217.598641] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing check_logdir /tmp/testlogs/ [ 219.899957] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing yml_node [ 222.814686] Lustre: DEBUG MARKER: Client: 2.16.58.106 [ 224.657799] Lustre: DEBUG MARKER: MDS: 2.16.58.106 [ 226.467252] Lustre: DEBUG MARKER: OSS: 2.16.58.106 [ 227.558863] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 10 15:46:16 EDT 2025 [ 237.235534] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 238.058516] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 254.432574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 254.436541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 254.444917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 254.945078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 254.946086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 254.950604] Lustre: Skipped 1 previous similar message [ 254.955103] Lustre: Skipped 2 previous similar messages [ 260.065123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 260.067958] Lustre: Skipped 3 previous similar messages [ 265.183131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 265.183676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 265.193297] Lustre: Skipped 3 previous similar messages [ 265.233374] LustreError: 12184:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 265.271900] Lustre: server umount lustre-MDT0000 complete [ 267.338356] LustreError: 6478:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760125616 with bad export cookie 3415603842277418991 [ 267.341521] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 267.342730] LustreError: 6478:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 270.308601] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 270.309276] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 270.314684] Lustre: Skipped 2 previous similar messages [ 273.494696] LustreError: 12388:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 273.497629] LustreError: 12388:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 273.670917] Lustre: server umount lustre-MDT0001 complete [ 276.276095] LustreError: 12589:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 276.282505] LustreError: 12589:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 276.308723] Lustre: server umount lustre-OST0000 complete [ 278.780464] LustreError: 12789:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 278.783401] LustreError: 12789:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 278.886597] Lustre: server umount lustre-OST0001 complete [ 282.980565] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 288.405801] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 294.149607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 297.763618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 301.160755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 305.276242] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 309.439882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 309.486153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 309.613362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 309.636581] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 309.681648] Lustre: lustre-MDT0000: new disk, initializing [ 309.768957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 309.779963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 311.957449] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 319.134663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 319.186216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 319.244523] Lustre: 14970:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 319.284827] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 319.288320] Lustre: Skipped 1 previous similar message [ 319.333471] Lustre: lustre-MDT0001: new disk, initializing [ 319.409045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 319.437132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 319.447083] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 321.561114] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 325.973286] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 328.212426] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 332.131254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 332.174253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 334.108336] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 334.167263] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 335.075444] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 339.344850] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 341.391198] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 341.517043] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 343.700462] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 343.821391] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 349.665910] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 349.669232] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 349.675423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 349.678047] Lustre: Skipped 4 previous similar messages [ 359.391137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 359.419147] LustreError: 17910:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 359.421166] LustreError: 17910:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 359.506717] Lustre: server umount lustre-OST0000 complete [ 365.027342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 365.037841] Lustre: Skipped 2 previous similar messages [ 367.438733] LustreError: 18113:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 367.442485] LustreError: 18113:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 367.495295] Lustre: server umount lustre-MDT0000 complete [ 369.585406] LustreError: 14963:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760125718 with bad export cookie 3415603842277420517 [ 369.588027] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 369.592833] LustreError: 14963:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 369.800769] Lustre: server umount lustre-MDT0001 complete [ 372.766657] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 15:48:41 (1760125721) [ 376.241789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 376.542180] LustreError: 18840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 376.563762] LustreError: 18840:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 376.603547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 376.607905] Lustre: Skipped 1 previous similar message [ 378.667422] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 381.921481] LustreError: 18840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 383.434530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 385.722672] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 388.710802] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 390.703539] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 394.348467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 397.434718] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 400.162566] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 411.668917] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 417.874943] LustreError: 21160:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 417.878163] LustreError: 21160:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 417.930454] Lustre: server umount lustre-MDT0000 complete [ 419.303995] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 419.308510] LustreError: 18835:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 419.314225] Lustre: Skipped 1 previous similar message [ 419.324760] LustreError: 18835:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 420.422308] LustreError: 18820:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760125769 with bad export cookie 3415603842277421406 [ 420.424350] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 420.430555] LustreError: 18820:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 420.671319] Lustre: server umount lustre-MDT0001 complete [ 432.427991] Lustre: server umount lustre-OST0000 complete [ 443.626172] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 449.484055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 449.724077] LustreError: 22887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 449.770605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 449.774406] Lustre: Skipped 2 previous similar messages [ 452.071937] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 455.139262] LustreError: 22887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 457.322921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 459.818973] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 461.604393] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 465.371555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 468.786418] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 473.603354] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 473.641364] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 473.720691] Lustre: lustre-OST0001: new disk, initializing [ 473.722442] Lustre: Skipped 1 previous similar message [ 473.725259] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 473.728210] Lustre: Skipped 1 previous similar message [ 475.507741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 475.514823] Lustre: Skipped 1 previous similar message [ 475.518115] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 475.585223] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 477.269261] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 483.815216] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 485.970907] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 498.660507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 498.663995] LustreError: Skipped 1 previous similar message [ 498.666080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 498.671717] Lustre: Skipped 1 previous similar message [ 498.677197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 498.680895] Lustre: Skipped 4 previous similar messages [ 500.178384] LustreError: 26367:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 500.181027] LustreError: 26367:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 501.217947] LustreError: 22887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 501.226054] LustreError: 22887:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 502.277591] Lustre: server umount lustre-MDT0000 complete [ 504.363564] LustreError: 23992:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760125853 with bad export cookie 3415603842277422246 [ 504.369348] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 504.370017] LustreError: 23992:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 504.596404] Lustre: server umount lustre-MDT0001 complete [ 517.077252] Lustre: server umount lustre-OST0000 complete [ 529.371479] Lustre: server umount lustre-OST0001 complete [ 537.941236] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 543.283503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 543.495863] LustreError: 28143:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 543.557876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 543.560401] Lustre: Skipped 3 previous similar messages [ 545.373458] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 549.697470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 551.600812] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 552.848335] Lustre: 29258:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 555.757053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 558.226329] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 562.137585] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 564.819685] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 573.856878] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 575.414096] Lustre: 31116:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 583.648981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 583.653976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 583.655829] Lustre: Skipped 5 previous similar messages [ 583.658570] Lustre: Skipped 1 previous similar message [ 588.546603] LustreError: 31319:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 588.550266] LustreError: 31319:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 588.591176] Lustre: server umount lustre-MDT0000 complete [ 588.769538] LustreError: 28142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 588.776800] LustreError: 28142:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 590.154985] LustreError: 28123:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760125938 with bad export cookie 3415603842277423548 [ 590.158017] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 590.163643] LustreError: 28123:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 590.341787] Lustre: server umount lustre-MDT0001 complete [ 602.118721] Lustre: server umount lustre-OST0000 complete [ 614.071477] Lustre: server umount lustre-OST0001 complete [ 617.700623] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 15:52:46 (1760125966) [ 629.254042] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 633.368186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 635.175880] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 638.758608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 640.465466] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 641.543262] Lustre: 35016:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 644.130718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 646.382057] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 649.859907] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 652.226432] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 666.228947] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 667.643044] Lustre: 36894:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 674.786976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 674.788132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 674.789498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 674.789507] Lustre: Skipped 2 previous similar messages [ 674.800037] Lustre: Skipped 3 previous similar messages [ 680.342339] Lustre: server umount lustre-MDT0000 complete [ 681.849741] LustreError: 34633:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760126030 with bad export cookie 3415603842277424654 [ 681.852968] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 681.854751] LustreError: 34633:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 682.016361] Lustre: server umount lustre-MDT0001 complete [ 694.272852] Lustre: server umount lustre-OST0000 complete [ 706.169596] Lustre: server umount lustre-OST0001 complete [ 709.560135] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 15:54:18 (1760126058) [ 715.916397] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 720.368907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 720.555472] LustreError: 39015:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 720.564831] LustreError: 39015:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 720.592496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 720.596975] Lustre: Skipped 7 previous similar messages [ 722.305929] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 726.039418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 728.102211] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 729.467446] Lustre: 40131:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 732.470975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 735.478798] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 739.951612] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 742.896939] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 752.180943] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 759.165387] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 759.168164] Lustre: Skipped 1 previous similar message [ 766.465726] LustreError: 42190:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 766.468591] LustreError: 42190:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 766.506838] Lustre: server umount lustre-MDT0000 complete [ 768.135153] LustreError: 38994:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760126116 with bad export cookie 3415603842277425760 [ 768.137604] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 768.140195] LustreError: 38994:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 774.370687] Lustre: server umount lustre-MDT0001 complete [ 778.298033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 793.056611] LustreError: 3628:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa0da39ef9c00 x1845625298953856/t0(0) o250->MGC192.168.204.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 794.952894] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 798.568727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 800.407770] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 802.720361] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 802.853787] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 806.585598] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 808.147922] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 810.911636] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 813.027072] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 813.028438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 813.032166] Lustre: Skipped 10 previous similar messages [ 813.036219] Lustre: Skipped 24 previous similar messages [ 818.593304] Lustre: server umount lustre-MDT0000 complete [ 820.048244] LustreError: 38995:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760126168 with bad export cookie 3415603842277426936 [ 820.052802] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 820.054084] LustreError: 38995:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 820.212929] Lustre: server umount lustre-MDT0001 complete [ 831.498446] Lustre: server umount lustre-OST0000 complete [ 843.332897] Lustre: server umount lustre-OST0001 complete [ 847.392572] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 15:56:36 (1760126196) [ 854.452651] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 858.810207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 858.996127] LustreError: 46352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 859.006231] LustreError: 46352:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 860.750400] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 864.424968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 866.304097] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 867.565320] Lustre: 47472:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 867.568797] Lustre: 47472:0:(mgs_llog.c:1345:mgs_modify_param()) Skipped 1 previous similar message [ 870.495035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 873.191712] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 876.875815] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 879.395669] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 888.351887] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 895.376633] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 906.856505] Lustre: server umount lustre-MDT0000 complete [ 908.465577] LustreError: 49335:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760126257 with bad export cookie 3415603842277427720 [ 908.469454] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 908.469988] LustreError: 49335:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 908.629503] Lustre: server umount lustre-MDT0001 complete [ 920.649252] Lustre: server umount lustre-OST0000 complete [ 931.522448] Lustre: server umount lustre-OST0001 complete [ 935.451399] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 15:58:04 (1760126284) [ 936.454029] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 937.221607] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 15:58:05 (1760126285) [ 941.353572] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 944.915698] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 949.155399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 951.750359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 954.564538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 954.598521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 954.719202] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 954.767606] Lustre: lustre-MDT0000: new disk, initializing [ 954.815677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 956.412183] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 960.060703] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 962.785529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 962.816545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 964.053659] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 964.071302] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 965.298409] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 968.920855] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 969.765220] Lustre: DEBUG MARKER: create test files [ 984.545449] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 988.227322] Lustre: server umount lustre-OST0000 complete [ 990.044745] Lustre: server umount lustre-MDT0000 complete [ 994.262167] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 995.547496] Key type lgssc unregistered [ 995.693285] LNet: 54883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 995.696617] LNetError: 54883:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 995.708504] LNet: Removed LNI 192.168.204.112@tcp [ 996.135289] Key type .llcrypt unregistered [ 996.136919] Key type ._llcrypt unregistered [ 1008.011311] Key type ._llcrypt registered [ 1008.012765] Key type .llcrypt registered [ 1008.058506] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 1008.485763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1008.500191] alg: No test for adler32 (adler32-zlib) [ 1009.365866] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 1009.467490] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 1011.055197] Key type lgssc registered [ 1011.489033] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1014.530649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1014.539871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1015.741644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1017.194492] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1019.257638] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1021.854343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1021.996921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1024.145708] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1026.085568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1026.121554] Lustre: 57465:0:(osd_internal.h:1341:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1026.125648] Lustre: 57465:0:(osd_handler.c:2078:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1026.129075] Lustre: 57465:0:(osd_handler.c:2085:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1026.132723] Lustre: 57465:0:(osd_handler.c:2095:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 1026.136756] Lustre: 57465:0:(osd_handler.c:2102:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1026.139615] Lustre: 57465:0:(osd_handler.c:2109:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1026.226433] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1029.500824] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1031.607882] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1036.256902] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1036.256903] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1036.257597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1039.921094] LustreError: 58208:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1040.001653] Lustre: server umount lustre-OST0000 complete [ 1041.540427] LustreError: 58410:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1041.543463] LustreError: 58410:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1041.638058] Lustre: server umount lustre-MDT0000 complete [ 1045.781553] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 1046.981781] Key type lgssc unregistered [ 1047.124836] LNet: 58943:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.127984] LNetError: 58943:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.137633] LNet: Removed LNI 192.168.204.112@tcp [ 1047.475148] Key type .llcrypt unregistered [ 1047.476487] Key type ._llcrypt unregistered [ 1060.173038] Key type ._llcrypt registered [ 1060.174670] Key type .llcrypt registered [ 1060.219572] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 1060.608041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1060.666039] alg: No test for adler32 (adler32-zlib) [ 1061.542637] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 1061.647519] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 1063.239483] Key type lgssc registered [ 1063.756121] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1068.643363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1068.665800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1070.263506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1075.732180] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1085.484476] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1099.895042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1100.665667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1109.042405] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1116.036697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1116.189831] Lustre: 61625:0:(osd_internal.h:1341:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1116.209108] Lustre: 61625:0:(osd_handler.c:2078:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1116.233871] Lustre: 61625:0:(osd_handler.c:2085:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1116.251233] Lustre: 61625:0:(osd_handler.c:2095:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 1116.272874] Lustre: 61625:0:(osd_handler.c:2102:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1116.285500] Lustre: 61625:0:(osd_handler.c:2109:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1116.934715] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1122.577565] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1162.211142] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1162.218768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.230336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1167.329793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1167.682138] LustreError: 62358:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1167.803088] Lustre: server umount lustre-OST0000 complete [ 1171.229901] LustreError: 62561:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1171.240656] LustreError: 62561:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1171.469962] Lustre: server umount lustre-MDT0000 complete [ 1179.799355] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 1182.300373] Key type lgssc unregistered [ 1182.544452] LNet: 63092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1182.556885] LNetError: 63092:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1183.591855] LNet: Removed LNI 192.168.204.112@tcp [ 1184.491640] Key type .llcrypt unregistered [ 1184.492980] Key type ._llcrypt unregistered [ 1198.710164] Key type ._llcrypt registered [ 1198.711758] Key type .llcrypt registered [ 1198.807553] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 1210.270745] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 1211.068251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1211.088288] alg: No test for adler32 (adler32-zlib) [ 1212.173670] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 1212.355380] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 1214.023318] Key type lgssc registered [ 1215.021949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1222.314158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1228.136240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1233.569916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1238.611250] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1244.708868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1244.756227] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1244.769460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1245.938265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1245.966535] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1246.025180] Lustre: lustre-MDT0000: new disk, initializing [ 1246.128244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1246.139817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1249.163571] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1257.803730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1257.863352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1257.981256] Lustre: 66197:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1258.044735] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1258.047561] Lustre: Skipped 1 previous similar message [ 1258.137740] Lustre: lustre-MDT0001: new disk, initializing [ 1258.279593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1258.332873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1258.356696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1261.865501] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1268.719154] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1271.681580] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1277.097765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1277.182056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1277.370336] Lustre: lustre-OST0000: new disk, initializing [ 1277.375256] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1277.452129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1282.166414] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1287.090169] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1287.715027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1287.719478] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1287.790220] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1289.427644] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1289.582202] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1292.797562] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1293.035833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1297.888218] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1297.894451] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1297.909359] LustreError: Skipped 1 previous similar message [ 1297.917522] Lustre: Skipped 1 previous similar message [ 1297.938833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1297.957564] Lustre: Skipped 1 previous similar message [ 1300.744729] LustreError: 69135:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1300.935608] Lustre: server umount lustre-OST0000 complete [ 1308.131493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1308.156652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1313.249357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1313.251948] Lustre: Skipped 1 previous similar message [ 1317.348061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1317.355921] Lustre: Skipped 1 previous similar message [ 1319.392627] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1319.490809] LustreError: 69337:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1319.501453] LustreError: 69337:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1319.622820] Lustre: server umount lustre-MDT0000 complete [ 1322.662832] LustreError: 66189:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760126671 with bad export cookie 10413741555509718053 [ 1322.665474] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1322.687858] LustreError: 66189:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1322.799854] LustreError: 69541:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1322.813626] LustreError: 69541:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1323.283203] Lustre: server umount lustre-MDT0001 complete [ 1330.886643] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 16:04:38 (1760126678) [ 1339.746309] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 1352.263906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1382.261315] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 1391.413814] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 1402.226110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1409.640752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1416.378740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1423.439193] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1430.869726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1430.945470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1431.176448] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1431.208974] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1431.276934] Lustre: lustre-MDT0000: new disk, initializing [ 1431.423683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1431.468962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1435.552539] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1446.531700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1446.613988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1446.723605] Lustre: 73781:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1446.798810] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1446.804827] Lustre: Skipped 1 previous similar message [ 1446.952968] Lustre: lustre-MDT0001: new disk, initializing [ 1447.110858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1447.173508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1447.189361] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1451.390941] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1459.523807] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1464.014892] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1471.342756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1471.461603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1471.845573] Lustre: lustre-OST0000: new disk, initializing [ 1471.853940] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1471.980232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1478.754555] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1480.241059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1480.251932] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1480.413065] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1486.886327] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1491.088075] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1491.303268] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1494.458256] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1494.678057] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1501.153199] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1501.164033] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1501.179513] LustreError: Skipped 1 previous similar message [ 1501.183020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1501.238086] Lustre: Skipped 2 previous similar messages [ 1502.596050] LustreError: 76720:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1502.597965] LustreError: 76720:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1502.854253] Lustre: server umount lustre-OST0000 complete [ 1508.836768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1508.848163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1511.394679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1511.409051] Lustre: Skipped 3 previous similar messages [ 1513.317577] LustreError: 76923:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1513.327179] LustreError: 76923:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1513.578581] Lustre: server umount lustre-MDT0000 complete [ 1516.525747] LustreError: 73792:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1516.537655] LustreError: 73792:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1518.282509] LustreError: 73772:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760126867 with bad export cookie 10413741555509718935 [ 1518.287522] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1518.301950] LustreError: 73772:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1518.836370] Lustre: server umount lustre-MDT0001 complete [ 1528.249550] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 16:07:56 (1760126876) [ 1535.316434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1535.758564] LustreError: 77796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1535.883421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1540.349325] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1541.097663] LustreError: 77796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1546.209807] LustreError: 77796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1550.618336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1555.554131] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1561.521167] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1565.647311] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1573.357480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1573.875332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1573.877577] Lustre: Skipped 1 previous similar message [ 1579.713268] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1585.049925] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1588.823743] Lustre: 80134:0:(genops.c:1791:obd_export_evict_by_uuid()) MGS: evicting 4293728e-26d7-4cb7-aca4-fbca31de6b01 at adminstrative request [ 1592.289669] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1592.302057] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 1602.532452] Lustre: MGS: Client 4293728e-26d7-4cb7-aca4-fbca31de6b01 (at 0@lo) reconnecting [ 1602.549186] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x90850ac536e17309 to 0x90850ac536e175a9 [ 1602.557290] Lustre: MGC192.168.204.112@tcp: Connection restored to 0@lo (at 0@lo) [ 1624.525036] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1624.583932] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1624.680767] Lustre: lustre-OST0001: new disk, initializing [ 1624.686098] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1630.532444] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1633.400105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1633.410037] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1633.533857] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1637.957987] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1644.000700] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1644.008595] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1644.027981] LustreError: Skipped 1 previous similar message [ 1644.031671] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1644.033728] Lustre: Skipped 2 previous similar messages [ 1644.043375] Lustre: Skipped 1 previous similar message [ 1655.263654] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1655.295081] LustreError: 81271:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 1655.301483] LustreError: 81271:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1655.376870] Lustre: server umount lustre-OST0001 complete [ 1659.367817] LustreError: 79395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1659.374974] LustreError: 79395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1662.952788] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1662.960580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1665.971513] Lustre: server umount lustre-OST0000 complete [ 1674.210782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1674.216768] LustreError: Skipped 1 previous similar message [ 1676.441345] LustreError: 81677:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1676.444450] LustreError: 81677:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1676.522777] Lustre: server umount lustre-MDT0000 complete [ 1679.841636] LustreError: 77796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1679.870381] LustreError: 77796:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1680.431778] LustreError: 77776:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760127029 with bad export cookie 10413741555509720489 [ 1680.463606] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1680.894152] Lustre: server umount lustre-MDT0001 complete [ 1689.925471] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 16:10:37 (1760127037) [ 1691.578977] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 1693.652563] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 16:10:41 (1760127041) [ 1695.310548] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 1697.351430] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 16:10:45 (1760127045) [ 1698.876288] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 1700.697687] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 16:10:48 (1760127048) [ 1702.634539] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 1707.778077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1708.855751] loop: module loaded [ 1708.873901] loop0: detected capacity change from 0 to 4294967296 [ 1708.905654] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 1710.578855] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 1713.454228] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 1717.541875] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 1726.554055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1730.811332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1734.382323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1738.458036] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1744.962712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1745.013491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1745.034353] LustreError: 84456:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-28) [ 1745.039059] LustreError: 84456:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 1745.043230] LustreError: 84456:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 1745.047811] LustreError: 84456:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 1763.402464] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 1773.636107] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 1784.525261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1791.639677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1798.243205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1804.785033] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1811.493162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1811.580814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1811.753332] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1811.791830] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1811.841463] Lustre: lustre-MDT0000: new disk, initializing [ 1811.952801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1811.962284] Lustre: Skipped 1 previous similar message [ 1811.978869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1816.112673] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1826.184324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1826.247258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1826.325785] Lustre: 87300:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1826.712921] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1830.603867] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1837.591414] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1841.007735] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1846.558397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1846.610413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1852.276028] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1852.498240] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1852.586810] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1859.088548] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1862.975841] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1863.180111] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1866.787494] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1866.997239] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1873.376034] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1873.379499] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1873.383161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1873.383176] Lustre: Skipped 10 previous similar messages [ 1873.410476] Lustre: Skipped 4 previous similar messages [ 1883.103160] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1883.131143] LustreError: 90239:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1883.133625] LustreError: 90239:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 1883.293790] Lustre: server umount lustre-OST0000 complete [ 1892.423028] Lustre: server umount lustre-MDT0000 complete [ 1893.858239] LustreError: 87311:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1893.865846] LustreError: 87311:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1895.169151] LustreError: 87293:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760127243 with bad export cookie 10413741555509721084 [ 1895.174860] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1895.180197] LustreError: 87293:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1895.446725] Lustre: server umount lustre-MDT0001 complete [ 1904.134332] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 16:14:12 (1760127252) [ 1910.827171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1914.900554] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1921.752847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1925.108592] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1930.149728] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1933.250314] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1938.499958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1943.658748] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1948.428434] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1951.752159] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1957.323088] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 1961.099982] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1961.294544] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1967.196839] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1967.297962] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1967.414761] Lustre: lustre-OST0001: new disk, initializing [ 1967.416798] Lustre: Skipped 2 previous similar messages [ 1967.419174] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1967.424914] Lustre: Skipped 3 previous similar messages [ 1967.510771] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1967.518552] Lustre: Skipped 5 previous similar messages [ 1973.282092] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1974.857753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1974.861850] Lustre: Skipped 2 previous similar messages [ 1974.869724] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1974.976810] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1980.537895] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1985.715622] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1986.035586] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1991.051835] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1991.366798] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2009.719177] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2035.573908] Lustre: Failing over lustre-MDT0000 [ 2035.897660] LustreError: 96262:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2035.901906] LustreError: 96262:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2035.955546] Lustre: server umount lustre-MDT0000 complete [ 2036.709840] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2036.716168] Lustre: Skipped 2 previous similar messages [ 2036.720399] LustreError: 92523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2036.727347] LustreError: 92523:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2053.087202] Lustre: 64412:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760127385/real 1760127385] req@0000000036cb8e94 x1845626469086848/t0(0) o400->MGC192.168.204.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760127401 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2053.143211] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2059.712696] LDISKFS-fs (dm-0): recovery complete [ 2059.716398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2062.317133] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x90850ac536e24008 [ 2062.327213] Lustre: MGC192.168.204.112@tcp: Connection restored to 0@lo (at 0@lo) [ 2062.605936] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2065.121352] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2065.129808] Lustre: 95564:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2066.165445] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2067.947107] Lustre: 91314:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2067.953245] Lustre: 91314:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 1 previous similar message [ 2067.963253] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2068.042560] LustreError: 97013:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2068.370760] LustreError: 97013:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2068.712691] LustreError: 97013:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2068.721400] LustreError: 97013:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 1 previous similar message [ 2069.039142] LustreError: 97013:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2069.048670] LustreError: 97013:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 1 previous similar message [ 2069.059588] Lustre: 97013:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2069.083276] Lustre: 97013:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 7 previous similar messages [ 2069.743973] LustreError: 97013:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2069.757988] LustreError: 97013:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 2 previous similar messages [ 2070.088153] LustreError: 97013:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2070.095755] LustreError: 97013:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 2 previous similar messages [ 2071.063948] Lustre: 97013:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2071.080916] Lustre: 97013:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 5 previous similar messages [ 2071.747102] LustreError: 97013:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2071.767437] LustreError: 97013:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 5 previous similar messages [ 2072.399770] LustreError: 97013:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2072.420199] LustreError: 97013:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 6 previous similar messages [ 2075.301341] Lustre: 97013:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2075.318706] Lustre: 97013:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 12 previous similar messages [ 2075.971846] LustreError: 97013:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2075.985865] LustreError: 97013:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 12 previous similar messages [ 2076.615154] LustreError: 97013:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2076.621717] LustreError: 97013:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 12 previous similar messages [ 2083.312318] Lustre: 97013:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2083.336812] Lustre: 97013:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 24 previous similar messages [ 2083.989406] LustreError: 97013:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2084.000588] LustreError: 97013:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 24 previous similar messages [ 2084.639344] LustreError: 97013:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2084.654941] LustreError: 97013:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 24 previous similar messages [ 2099.528191] Lustre: 97013:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2099.540035] Lustre: 97013:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 50 previous similar messages [ 2100.189925] LustreError: 97013:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2100.204722] LustreError: 97013:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 50 previous similar messages [ 2100.847164] LustreError: 97013:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2100.853019] LustreError: 97013:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 50 previous similar messages [ 2125.215789] Lustre: 97013:0:(ldlm_lib.c:2375:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2125.225953] Lustre: 97013:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2125.241044] Lustre: 97013:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 3af29685-3f63-4463-8dbd-0a8ccffe7131@192.168.204.12@tcp [ 2125.248417] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2125.251994] Lustre: 97013:0:(ldlm_lib.c:1897:abort_req_replay_queue()) @@@ aborted: req@000000004b03ecfb x1845626457362304/t0(8589934776) o35->3af29685-3f63-4463-8dbd-0a8ccffe7131@192.168.204.12@tcp:514/0 lens 392/0 e 0 to 0 dl 1760127484 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2125.266959] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2125.279202] Lustre: lustre-MDT0000: Denying connection for new client 3af29685-3f63-4463-8dbd-0a8ccffe7131 (at 192.168.204.12@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) already passed deadline 0:00 [ 2125.279543] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2125.341205] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2125.343830] Lustre: Skipped 2 previous similar messages [ 2125.348797] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2125.397341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:79 to 0x280000401:577) [ 2125.400937] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:47 to 0x2c0000401:545) [ 2139.626786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2139.638659] Lustre: Skipped 1 previous similar message [ 2139.641924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2139.644065] Lustre: Skipped 5 previous similar messages [ 2150.881058] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2151.028776] Lustre: server umount lustre-OST0000 complete [ 2154.984970] LustreError: 92911:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2154.991749] LustreError: 92911:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 34 previous similar messages [ 2159.586195] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2159.592492] LustreError: Skipped 2 previous similar messages [ 2160.824089] Lustre: server umount lustre-OST0001 complete [ 2170.028020] LustreError: 98079:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 2170.037430] LustreError: 98079:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 2170.141694] Lustre: server umount lustre-MDT0000 complete [ 2173.509627] LustreError: 91296:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760127522 with bad export cookie 10413741555509772296 [ 2173.512645] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2173.524887] LustreError: 91296:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2174.015743] Lustre: server umount lustre-MDT0001 complete [ 2181.804909] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 16:18:49 (1760127529) [ 2188.398270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2188.405408] Lustre: *** cfs_fail_loc=197, val=0*** [ 2203.999757] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2209.122769] LustreError: 98972:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.112@tcp: failed processing log, type 4: rc = -110 [ 2214.278862] Lustre: *** cfs_fail_loc=197, val=0*** [ 2214.283112] Lustre: Skipped 65 previous similar messages [ 2234.847306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2234.849591] Lustre: Skipped 1 previous similar message [ 2239.810418] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2244.779251] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2246.835359] Lustre: server umount lustre-OST0000 complete [ 2254.900144] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 16:20:02 (1760127602) [ 2275.613933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2280.001866] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2285.539455] LustreError: 100482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2285.574690] LustreError: 100482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2288.314648] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2292.854740] LustreError: 100513:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2292.869623] LustreError: 100513:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2292.885205] LustreError: 100513:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 2293.403685] Lustre: server umount lustre-MDT0000 complete [ 2309.096921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2325.336494] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 2335.057570] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 2345.411689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2351.102437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2356.357195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2362.353604] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2367.842665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2367.937104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2368.181930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2368.216451] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2368.267104] Lustre: lustre-MDT0000: new disk, initializing [ 2368.431268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2372.552306] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2382.352358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2382.411515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2382.486099] Lustre: 104553:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2382.722753] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2386.019340] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2392.951331] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2396.665316] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2402.234450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2402.338559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2407.759501] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2411.641147] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2414.478509] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2418.289987] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2418.513864] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2422.184235] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2422.354195] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2426.851332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2426.854809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2426.869543] Lustre: Skipped 5 previous similar messages [ 2426.884800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2426.892361] Lustre: Skipped 10 previous similar messages [ 2438.111138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2438.140302] LustreError: 107494:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2438.146196] LustreError: 107494:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 2438.280949] Lustre: server umount lustre-OST0000 complete [ 2447.680728] Lustre: server umount lustre-MDT0000 complete [ 2450.541257] LustreError: 104544:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760127799 with bad export cookie 10413741555509782936 [ 2450.547105] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2450.561817] LustreError: 104544:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2450.935910] Lustre: server umount lustre-MDT0001 complete [ 2457.724906] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 16:23:25 (1760127805) [ 2467.269344] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 2469.722718] Key type lgssc unregistered [ 2469.952400] LNet: 108624:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2469.967699] LNetError: 108624:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2469.981915] LNet: Removed LNI 192.168.204.112@tcp [ 2470.763279] Key type .llcrypt unregistered [ 2470.765081] Key type ._llcrypt unregistered [ 2484.797692] Key type ._llcrypt registered [ 2484.800223] Key type .llcrypt registered [ 2484.917452] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 2496.499554] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 2496.986974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2497.055145] alg: No test for adler32 (adler32-zlib) [ 2498.020743] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 2498.196329] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 2499.824195] Key type lgssc registered [ 2500.621708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2506.493518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2510.777819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2515.000331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2519.774788] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2524.757131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2529.833900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2529.864737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2529.876335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2531.036910] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2531.061143] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2531.105417] Lustre: lustre-MDT0000: new disk, initializing [ 2531.171264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2531.222497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2533.802682] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2542.656487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2542.731872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2542.902869] Lustre: lustre-OST0000: new disk, initializing [ 2542.906971] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2542.909508] Lustre: Skipped 1 previous similar message [ 2542.973443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2543.219443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2543.227780] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2543.250983] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2547.094882] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2555.125234] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2555.176403] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2555.286613] Lustre: lustre-OST0001: new disk, initializing [ 2555.295405] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2555.352597] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2559.133976] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2563.187458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 2563.192562] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 2563.211993] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 2568.160989] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2568.162882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2568.166066] Lustre: Skipped 1 previous similar message [ 2568.170956] Lustre: Skipped 1 previous similar message [ 2571.537759] LustreError: 114179:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 2571.720493] Lustre: server umount lustre-MDT0000 complete [ 2576.226766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2579.552990] LustreError: 113587:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760127928 with bad export cookie 15824796052905115680 [ 2579.555459] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2579.559791] LustreError: 113587:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2579.588466] LustreError: 114774:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 2579.594107] LustreError: 114774:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2579.620973] Lustre: server umount lustre-OST0000 complete [ 2582.158350] LustreError: 114976:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2582.165348] LustreError: 114976:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2582.300385] Lustre: server umount lustre-OST0001 complete [ 2588.192478] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 16:25:36 (1760127936) [ 2594.117833] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 2599.055913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2599.709577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2603.556607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2604.537301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2609.909352] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 16:25:58 (1760127958) [ 2617.791641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2619.061385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2620.430532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2621.749707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2622.453048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2623.899886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2637.655358] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 2644.442457] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 2652.316714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2657.925949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2662.898852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2668.154110] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2674.647863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2674.721808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2674.922392] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2674.955728] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2675.016703] Lustre: lustre-MDT0000: new disk, initializing [ 2675.095707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2675.112136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2680.478746] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2692.999620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2693.086855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2693.205578] Lustre: 120297:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2693.309715] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2693.328195] Lustre: Skipped 1 previous similar message [ 2693.391613] Lustre: lustre-MDT0001: new disk, initializing [ 2693.521403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2693.567958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2693.578637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2697.774930] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2704.227861] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2707.647297] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2713.490856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2713.539227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2713.730713] Lustre: lustre-OST0000: new disk, initializing [ 2713.737648] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2713.814903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2717.899222] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2719.321501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2719.330585] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2719.421858] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2724.828289] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2728.836451] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2729.034535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2733.082125] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2733.332452] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2740.194338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2740.202839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2741.421725] LustreError: 123241:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2741.432723] LustreError: 123241:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2741.554724] Lustre: server umount lustre-OST0000 complete [ 2745.314039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2745.320909] Lustre: Skipped 1 previous similar message [ 2745.324279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2745.327732] Lustre: Skipped 2 previous similar messages [ 2750.434273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2750.439697] Lustre: Skipped 1 previous similar message [ 2751.212163] LustreError: 123442:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2751.230336] LustreError: 123442:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2751.372488] Lustre: server umount lustre-MDT0000 complete [ 2754.554955] LustreError: 120291:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760128103 with bad export cookie 15824796052905116898 [ 2754.561570] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2754.565841] LustreError: 120291:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2754.990424] Lustre: server umount lustre-MDT0001 complete [ 2762.747724] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 16:28:30 (1760128110) [ 2768.142757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2768.585490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2772.080245] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2773.991071] LustreError: 124316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2774.005152] LustreError: 124316:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2779.109163] LustreError: 124316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2779.988397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2783.949622] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2788.578577] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2791.759894] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2797.745506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2803.811926] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2810.120932] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2819.896467] Lustre: *** cfs_fail_loc=159, val=0*** [ 2819.901024] Lustre: Skipped 3 previous similar messages [ 2836.286846] Lustre: lustre-MDT0000: Client 98dff940-6052-434e-8d29-12158fded519 (at 192.168.204.12@tcp) reconnecting [ 2838.098056] Lustre: *** cfs_fail_loc=159, val=0*** [ 2838.100033] Lustre: Skipped 2 previous similar messages [ 2853.632609] Lustre: lustre-MDT0000: Client 98dff940-6052-434e-8d29-12158fded519 (at 192.168.204.12@tcp) reconnecting [ 2859.384630] LustreError: 126805:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2859.387729] LustreError: 126805:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2859.555199] Lustre: server umount lustre-OST0000 complete [ 2863.010175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2863.013680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2863.019194] Lustre: Skipped 2 previous similar messages [ 2873.330750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2873.341784] Lustre: Skipped 4 previous similar messages [ 2876.191615] Lustre: 109944:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760128208/real 1760128208] req@0000000018443c9c x1845627816347392/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1760128224 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2876.213630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2876.895180] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2876.968813] LustreError: 127006:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2876.971541] LustreError: 127006:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2877.048348] Lustre: server umount lustre-MDT0000 complete [ 2877.410217] Lustre: 109946:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760128210/real 1760128210] req@0000000023b81f4e x1845627816347776/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1760128226 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2878.431120] Lustre: 109945:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760128211/real 1760128211] req@0000000079c312b4 x1845627816349440/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1760128227 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2880.288640] LustreError: 124297:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760128229 with bad export cookie 15824796052905117773 [ 2880.290813] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2880.297713] LustreError: 124297:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2880.558377] Lustre: server umount lustre-MDT0001 complete [ 2890.472327] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 2892.850823] Key type lgssc unregistered [ 2893.049625] LNet: 127741:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2893.055086] LNetError: 127741:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2893.062678] LNet: Removed LNI 192.168.204.112@tcp [ 2893.568403] Key type .llcrypt unregistered [ 2893.573895] Key type ._llcrypt unregistered [ 2912.770585] Key type ._llcrypt registered [ 2912.771964] Key type .llcrypt registered [ 2912.851415] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 2913.608679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2913.669796] alg: No test for adler32 (adler32-zlib) [ 2914.655293] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 2914.829427] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 2916.487172] Key type lgssc registered [ 2917.587254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2925.050790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2925.080436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2926.531675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2929.931754] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2931.684739] LustreError: 129118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2931.692296] LustreError: 129118:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2936.808737] LustreError: 129118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2938.100116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2938.422662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2941.534730] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2946.757983] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2949.687423] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2956.119951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2956.510768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2961.702223] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2967.566628] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2971.908903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 2980.813517] Lustre: *** cfs_fail_loc=159, val=0*** [ 2996.985766] Lustre: lustre-MDT0000: Client 7e94608b-8b08-45ce-8a11-0893168fb250 (at 192.168.204.12@tcp) reconnecting [ 2999.153980] Lustre: *** cfs_fail_loc=159, val=0*** [ 3015.464033] Lustre: lustre-MDT0001: Client 7e94608b-8b08-45ce-8a11-0893168fb250 (at 192.168.204.12@tcp) reconnecting [ 3016.789970] Lustre: *** cfs_fail_loc=159, val=0*** [ 3016.791542] Lustre: Skipped 3 previous similar messages [ 3033.350619] Lustre: lustre-MDT0001: Client 7e94608b-8b08-45ce-8a11-0893168fb250 (at 192.168.204.12@tcp) reconnecting [ 3048.394430] Lustre: 131845:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3050.607299] Lustre: *** cfs_fail_loc=159, val=0*** [ 3050.609219] Lustre: Skipped 4 previous similar messages [ 3067.114360] Lustre: lustre-MDT0000: Client 121f226f-7481-4245-8e8c-0765441c85cf (at 192.168.204.12@tcp) reconnecting [ 3068.096451] Lustre: *** cfs_fail_loc=159, val=0*** [ 3068.100229] Lustre: Skipped 16 previous similar messages [ 3084.534444] Lustre: lustre-MDT0000: Client 121f226f-7481-4245-8e8c-0765441c85cf (at 192.168.204.12@tcp) reconnecting [ 3085.422922] Lustre: 132132:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3105.252185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3105.252747] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3105.263034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3108.977108] LustreError: 132259:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3109.096498] Lustre: server umount lustre-OST0000 complete [ 3112.417888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3112.421308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3112.426811] Lustre: Skipped 1 previous similar message [ 3112.430229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3112.433093] Lustre: Skipped 1 previous similar message [ 3115.489266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3115.490212] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3117.410402] LustreError: 132460:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3117.418142] LustreError: 132460:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3117.532178] Lustre: server umount lustre-MDT0000 complete [ 3120.497915] LustreError: 129097:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760128469 with bad export cookie 17616301732899357494 [ 3120.503073] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3120.508018] LustreError: 129097:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3120.596400] LustreError: 132662:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3120.599350] LustreError: 132662:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3120.745518] Lustre: server umount lustre-MDT0001 complete [ 3127.983587] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 3129.944098] Key type lgssc unregistered [ 3130.125714] LNet: 133193:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3130.131292] LNetError: 133193:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3130.146457] LNet: Removed LNI 192.168.204.112@tcp [ 3130.691089] Key type .llcrypt unregistered [ 3130.697078] Key type ._llcrypt unregistered [ 3145.565215] Key type ._llcrypt registered [ 3145.566820] Key type .llcrypt registered [ 3145.627792] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 3146.059501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3146.121897] alg: No test for adler32 (adler32-zlib) [ 3147.064451] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 3147.198097] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 3148.823137] Key type lgssc registered [ 3149.493506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3155.002694] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3155.017867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3156.355069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3158.963384] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3164.061672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3164.264168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3166.666589] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3169.903436] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3171.869341] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3176.297348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3176.530253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3180.771386] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3181.708995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 3184.940889] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3188.844920] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 3193.099540] Lustre: 136818:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3194.879417] Lustre: 136866:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3196.090656] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 3196.812906] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 3199.050285] Lustre: 137014:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 3199.637166] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 3200.381658] Lustre: 137113:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3202.528127] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3202.531589] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3202.538257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3204.582179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3204.582372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3204.586075] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3208.634164] LustreError: 137210:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3208.829654] Lustre: server umount lustre-OST0000 complete [ 3214.819931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3214.834337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3214.834841] Lustre: Skipped 1 previous similar message [ 3214.837356] Lustre: Skipped 1 previous similar message [ 3217.886336] LustreError: 137418:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3217.889370] LustreError: 137418:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3217.979214] Lustre: server umount lustre-MDT0000 complete [ 3219.936311] LustreError: 134569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3219.947470] LustreError: 134569:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3220.542421] LustreError: 135288:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760128569 with bad export cookie 1894945681342792534 [ 3220.543873] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3220.547983] LustreError: 135288:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3220.643589] LustreError: 137621:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3220.652481] LustreError: 137621:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3220.842901] Lustre: server umount lustre-MDT0001 complete [ 3227.360306] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 3228.996890] Key type lgssc unregistered [ 3229.235571] LNet: 138152:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3229.248803] LNetError: 138152:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3229.261600] LNet: Removed LNI 192.168.204.112@tcp [ 3229.740299] Key type .llcrypt unregistered [ 3229.741982] Key type ._llcrypt unregistered [ 3244.113134] Key type ._llcrypt registered [ 3244.115341] Key type .llcrypt registered [ 3244.186337] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 3244.607865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3244.639207] alg: No test for adler32 (adler32-zlib) [ 3245.595168] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 3245.722544] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 3247.343177] Key type lgssc registered [ 3247.997307] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3251.778941] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3251.791374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3253.049958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3255.099792] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3259.858262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3260.082153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3262.135365] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3265.231562] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3267.367160] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3271.061472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3271.243610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3274.154078] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3277.416874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 3277.562529] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3284.599358] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 3290.774576] Lustre: *** cfs_fail_loc=159, val=0*** [ 3307.279228] Lustre: lustre-MDT0000: Client 221aa4e3-163f-4cd9-b530-ced285973915 (at 192.168.204.12@tcp) reconnecting [ 3310.562215] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3310.569695] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3310.570267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3310.576139] Lustre: Skipped 1 previous similar message [ 3315.680788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3315.684467] Lustre: Skipped 1 previous similar message [ 3315.891151] LustreError: 141922:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3316.025123] Lustre: server umount lustre-OST0000 complete [ 3320.800775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3320.802372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3320.807766] Lustre: Skipped 2 previous similar messages [ 3324.467621] LustreError: 142123:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3324.474722] LustreError: 142123:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3324.593362] Lustre: server umount lustre-MDT0000 complete [ 3325.928699] LustreError: 140239:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3325.942600] LustreError: 140239:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3327.349783] LustreError: 142124:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760128676 with bad export cookie 4466339611682061666 [ 3327.352515] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3327.363031] LustreError: 142124:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3327.450140] LustreError: 142327:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3327.454045] LustreError: 142327:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3327.597385] Lustre: server umount lustre-MDT0001 complete [ 3334.106444] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 3335.563081] Key type lgssc unregistered [ 3335.758587] LNet: 142858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3335.762209] LNetError: 142858:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3335.771777] LNet: Removed LNI 192.168.204.112@tcp [ 3336.226116] Key type .llcrypt unregistered [ 3336.227600] Key type ._llcrypt unregistered [ 3348.990499] Key type ._llcrypt registered [ 3348.993069] Key type .llcrypt registered [ 3349.053769] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 3349.596790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3349.610923] alg: No test for adler32 (adler32-zlib) [ 3350.520304] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 3350.649379] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 3352.255378] Key type lgssc registered [ 3352.977906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3358.057241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3358.075907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3359.369284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3362.030344] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3368.214060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3368.467884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3371.150585] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3374.495710] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3376.652861] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3380.613976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3380.851823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3384.542241] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3384.967267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 3388.121680] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3394.154262] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 3409.357524] Lustre: lustre-MDT0001: evicting 19abd7a2-aa92-4e4f-b8d6-aa3609c5786d (at 192.168.204.12@tcp) by administrative request [ 3409.364706] Lustre: Skipped 1 previous similar message [ 3435.489668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3435.495077] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3435.496954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3435.507608] Lustre: Skipped 1 previous similar message [ 3440.613658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3440.617853] Lustre: Skipped 2 previous similar messages [ 3441.393122] LustreError: 147020:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3441.534424] Lustre: server umount lustre-OST0000 complete [ 3445.220106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3445.223341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3445.232830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3450.320098] LustreError: 147221:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3450.322765] LustreError: 147221:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3450.377026] Lustre: server umount lustre-MDT0000 complete [ 3450.853131] LustreError: 146595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3450.863135] LustreError: 146595:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3452.539917] LustreError: 144214:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760128801 with bad export cookie 9178334360092876078 [ 3452.544476] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3452.548590] LustreError: 144214:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3452.637091] LustreError: 147422:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3452.639527] LustreError: 147422:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3452.768988] Lustre: server umount lustre-MDT0001 complete [ 3457.853768] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 3459.283181] Key type lgssc unregistered [ 3459.450811] LNet: 147954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3459.454654] LNetError: 147954:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3459.464443] LNet: Removed LNI 192.168.204.112@tcp [ 3459.863792] Key type .llcrypt unregistered [ 3459.866138] Key type ._llcrypt unregistered [ 3498.458530] Key type ._llcrypt registered [ 3498.460266] Key type .llcrypt registered [ 3498.506248] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 3499.036258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3499.060684] alg: No test for adler32 (adler32-zlib) [ 3499.967324] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 3500.100552] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 3501.703293] Key type lgssc registered [ 3502.374563] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3506.754231] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3506.770205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3508.105947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3510.622338] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3516.004371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3516.261296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3518.483419] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3521.127480] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3522.953966] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3526.464164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3526.655100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3529.919693] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3532.848329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 3532.858564] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3541.043129] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 3548.128220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3548.132734] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3548.139098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3551.201789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3551.203036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3551.206677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3553.394151] LustreError: 152681:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3553.477159] Lustre: server umount lustre-OST0000 complete [ 3556.320641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3556.327857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3556.330620] Lustre: Skipped 2 previous similar messages [ 3561.230320] LustreError: 152882:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3561.235019] LustreError: 152882:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3561.267315] Lustre: server umount lustre-MDT0000 complete [ 3561.440604] LustreError: 151124:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3561.448412] LustreError: 151124:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3563.063605] LustreError: 150379:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760128911 with bad export cookie 2043183191951152502 [ 3563.066036] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3563.071159] LustreError: 150379:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3563.149417] LustreError: 153085:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3563.151498] LustreError: 153085:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3563.269918] Lustre: server umount lustre-MDT0001 complete [ 3568.132393] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 3569.531118] Key type lgssc unregistered [ 3569.714838] LNet: 153616:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3569.720965] LNetError: 153616:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3569.732510] LNet: Removed LNI 192.168.204.112@tcp [ 3570.164802] Key type .llcrypt unregistered [ 3570.166374] Key type ._llcrypt unregistered [ 3582.404765] Key type ._llcrypt registered [ 3582.406411] Key type .llcrypt registered [ 3582.462223] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 3582.878791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3582.934065] alg: No test for adler32 (adler32-zlib) [ 3583.864674] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 3583.999783] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 3585.599200] Key type lgssc registered [ 3586.388817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3590.538555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3595.832285] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 16:42:24 (1760128944) [ 3606.178463] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 3612.383432] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 3619.382301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3623.933224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3627.250923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3630.580442] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3634.937333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3634.971123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3634.980566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3636.142410] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3636.168951] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3636.217458] Lustre: lustre-MDT0000: new disk, initializing [ 3636.290450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3636.313882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3638.889311] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3645.650546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3645.698409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3645.748636] Lustre: 157816:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3645.786759] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3645.789082] Lustre: Skipped 1 previous similar message [ 3645.843046] Lustre: lustre-MDT0001: new disk, initializing [ 3645.907709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3645.928148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3645.938036] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3648.394951] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3653.296509] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3655.518484] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3660.374371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3660.426164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3660.634624] Lustre: lustre-OST0000: new disk, initializing [ 3660.639427] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3660.694374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3661.337729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3661.345421] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3661.431667] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3664.094647] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3668.589691] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3676.642320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3676.645810] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3676.649060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3676.656519] Lustre: Skipped 1 previous similar message [ 3679.350952] LustreError: 160402:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3679.496962] Lustre: server umount lustre-OST0000 complete [ 3686.880542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3686.885062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3686.886165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3686.906332] Lustre: Skipped 2 previous similar messages [ 3688.283832] LustreError: 160605:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3688.295085] LustreError: 160605:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3688.362231] Lustre: server umount lustre-MDT0000 complete [ 3690.927990] LustreError: 159562:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760129039 with bad export cookie 732067947520446264 [ 3690.934838] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3690.940130] LustreError: 159562:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3691.041889] LustreError: 160806:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3691.049138] LustreError: 160806:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3691.235481] Lustre: server umount lustre-MDT0001 complete [ 3698.393413] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 3700.398654] Key type lgssc unregistered [ 3700.668700] LNet: 161338:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3700.674525] LNetError: 161338:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3700.690677] LNet: Removed LNI 192.168.204.112@tcp [ 3701.250742] Key type .llcrypt unregistered [ 3701.252614] Key type ._llcrypt unregistered [ 3716.624265] Key type ._llcrypt registered [ 3716.628398] Key type .llcrypt registered [ 3716.714377] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 3717.182798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3717.190583] alg: No test for adler32 (adler32-zlib) [ 3718.158948] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 3718.333695] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 3719.976475] Key type lgssc registered [ 3720.737654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3725.719799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3725.730452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3726.984745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3729.441385] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3734.985056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3735.238301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3737.707204] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3740.936888] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3743.000697] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3746.838670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3747.027672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3750.208350] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3752.173235] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3752.990397] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3762.110639] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3762.123455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3762.133294] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3762.137486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3762.147946] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3762.848607] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3762.869307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3762.876883] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3762.882071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3762.891036] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3763.483238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3764.054683] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3764.057933] Lustre: Skipped 1 previous similar message [ 3764.071567] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3764.081412] Lustre: Skipped 1 previous similar message [ 3764.087274] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3764.090101] Lustre: Skipped 1 previous similar message [ 3764.096798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3764.108933] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3764.118492] Lustre: Skipped 1 previous similar message [ 3764.645941] LustreError: 162730:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 3764.652297] LustreError: 162730:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 3764.655759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3765.296847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3765.887559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3766.471568] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3766.475738] Lustre: Skipped 3 previous similar messages [ 3766.492965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3766.502859] Lustre: Skipped 3 previous similar messages [ 3766.508562] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3766.512264] Lustre: Skipped 3 previous similar messages [ 3766.516638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3766.526862] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3766.531568] Lustre: Skipped 3 previous similar messages [ 3767.103487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3767.717702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3768.417313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3768.977096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3769.474031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3769.981513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3770.462527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3771.005875] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3771.009227] Lustre: Skipped 7 previous similar messages [ 3771.018088] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3771.023760] Lustre: Skipped 7 previous similar messages [ 3771.026873] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3771.031714] Lustre: Skipped 7 previous similar messages [ 3771.034603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3771.045368] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3771.051425] Lustre: Skipped 7 previous similar messages [ 3771.545635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3772.094163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3772.591453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3773.083640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3773.595477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3774.055868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3774.594337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3775.136347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3775.674423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3776.236688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3776.776832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3777.340606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3777.965901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3778.586923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3779.195363] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3779.199391] Lustre: Skipped 14 previous similar messages [ 3779.208780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3779.214556] Lustre: Skipped 14 previous similar messages [ 3779.217729] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3779.222391] Lustre: Skipped 14 previous similar messages [ 3779.224935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3779.234388] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3779.237194] Lustre: Skipped 14 previous similar messages [ 3779.810364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3780.406678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3781.047701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3781.612347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3782.168562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3782.707713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3783.288818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3783.937632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3784.511541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3785.178665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3785.719140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3786.303059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3786.925922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3787.522917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3788.129454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3788.663302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3789.202761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3789.751651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3790.329702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3790.940851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3791.571476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3792.172543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3792.836233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3793.453090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3794.030091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3794.562545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3795.120406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3795.607858] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3795.619252] Lustre: Skipped 27 previous similar messages [ 3795.632611] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3795.647660] Lustre: Skipped 27 previous similar messages [ 3795.654207] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3795.658514] Lustre: Skipped 27 previous similar messages [ 3795.665338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3795.675167] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3795.679408] Lustre: Skipped 27 previous similar messages [ 3796.252834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3796.800629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3797.305177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3797.796429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3798.312330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3798.759516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3799.296686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3799.808565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3800.350928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3800.861598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3801.378486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3801.841935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3802.304455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3802.753830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3803.204903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3803.622435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3804.089996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3804.547300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3805.001414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3805.440037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3805.904255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3806.429593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3806.907703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3807.423619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3807.956197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3808.445782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3808.929588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3809.449481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3809.923816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3810.468993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3810.990640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3811.577859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3812.143266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3812.623212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3813.105579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3813.612770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3814.132187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3814.562850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3815.027720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3815.497547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3815.905243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3816.354749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3816.781919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3817.240663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3817.633991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3818.045630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3818.466462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3818.886860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3819.329409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3819.776335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3820.229950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3820.706846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3821.126822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3821.529380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3821.938376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3822.385482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3822.900734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3823.368307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3823.772912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3824.177367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3824.624208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3825.059986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3825.454277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3825.934140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3826.366194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3826.860083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3827.402262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3827.877670] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3827.880461] Lustre: Skipped 67 previous similar messages [ 3827.888304] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3827.893575] Lustre: Skipped 67 previous similar messages [ 3827.895697] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3827.898753] Lustre: Skipped 67 previous similar messages [ 3827.901235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3827.906302] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3827.910195] Lustre: Skipped 67 previous similar messages [ 3828.401611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3828.872411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3829.349290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3829.809587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3830.231434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3830.644147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3831.067994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3831.511895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3831.921800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3832.392265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3832.813464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3833.263280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3833.748432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3834.183496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3834.585691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3835.070840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3835.493535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3835.965239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3836.493656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3836.986304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3837.449230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3837.934243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3838.406221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3838.922254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3839.424633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3840.054946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3840.572131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3841.073139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3841.630578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3842.187688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3842.726595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3843.270871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3843.839635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3844.425871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3845.053670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3845.602620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3846.123725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3846.611481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3847.128511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3847.634848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3848.099170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3848.608462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3849.056170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3849.576759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3850.066456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3850.567987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3851.083795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3851.636146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3852.107496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3852.567061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3853.116285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3853.654074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3854.176517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3854.691649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3855.179912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3855.682290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3856.145453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3856.681602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3857.160883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3857.660439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3858.253961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3858.796315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3859.330866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3859.897952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3860.422393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3861.020708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3861.601800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3862.195604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3862.796459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3863.432151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3864.015462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3864.624588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3865.244964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3865.858489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3866.415992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3866.992799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3867.623800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3868.140612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3868.698132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3869.198892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3869.719714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3870.278963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3870.869458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3871.429726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3871.911871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3872.373141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3872.864666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3873.429567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3873.946699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3874.449704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3874.947840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3875.475811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3876.030735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3876.640053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3877.207125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3877.776874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3878.362031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3878.951671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3879.513936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3880.133742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3880.613287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3881.201262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3881.696639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3882.239692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3882.716627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3883.165988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3883.613555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3884.049214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3884.512661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3884.981949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3885.522443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3885.969704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3886.410555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3886.886977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3887.376634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3887.834151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3888.267179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3888.724582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3889.180128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3889.633322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3890.154843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3890.764689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3891.343634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3891.881616] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3891.885671] Lustre: Skipped 123 previous similar messages [ 3891.896615] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3891.905087] Lustre: Skipped 123 previous similar messages [ 3891.909976] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3891.914896] Lustre: Skipped 123 previous similar messages [ 3891.916851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3891.924041] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3891.926734] Lustre: Skipped 123 previous similar messages [ 3892.487357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3893.063872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3893.582308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3894.208435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3894.664877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3895.130662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3895.678071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3896.215535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3896.762454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3897.433646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3898.167319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3898.858248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3899.553608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3900.182955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3900.836716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3901.447468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3902.078756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3902.676787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3903.259923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3903.913749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3904.602428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3905.187478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3905.830216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3906.424643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3906.983184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3907.616921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3908.209618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3908.760919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3909.389612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3909.953716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3910.530205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3911.101434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3911.645576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3912.220584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3912.833431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3913.405488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3913.878957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3914.442926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3914.979079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3915.592232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3916.202429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3916.745601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3917.344537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3917.913279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3918.483389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3919.123573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3919.713150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3920.321971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3920.924439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3921.550729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3922.088850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3922.698688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3923.355119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3923.931547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3924.504248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3925.054159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3925.604716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3926.131689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3926.758331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3927.506682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3928.095946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3928.771854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3929.417032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3930.020430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3930.679890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3931.342272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3931.897706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3932.472507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3933.086063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3933.699513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3934.188960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3934.767837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3935.320723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3935.914211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3936.489178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3937.176453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3937.797932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3938.514426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3939.199336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3940.019033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3940.686558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3941.427769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3942.070609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3942.778071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3943.451680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3944.128704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3944.821657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3945.545530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3946.254893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3946.935705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3947.631668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3948.447655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3949.174503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3949.874711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3950.563375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3951.131022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3951.772819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3952.410645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3953.085946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3953.786839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3954.519257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3955.222571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3955.933024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3956.618224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3957.258697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3957.802348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3958.380034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3958.949792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3959.465354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3959.967840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3960.539600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3961.025679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3961.561439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3962.154131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3962.694129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3963.245828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3963.816476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3964.402990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3964.986676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3965.643419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3966.194966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3966.887893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3967.489071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3968.150883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3968.887847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3969.500264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3970.119621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3970.782608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3971.388246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3972.076790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3972.663780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3973.354311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3973.906523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3974.508202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3975.071156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3975.695520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3976.386482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3977.048807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3977.665822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3978.255620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3978.845531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3979.492269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3980.090827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3980.702936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3981.236792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3981.811723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3982.388843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3982.985376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3983.518902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3984.033592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3984.642905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3985.213149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3985.790865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3986.431879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3986.948281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3987.498688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3988.127774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3988.801139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3989.525499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3990.158861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3990.864991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3991.539054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3992.127671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3992.763210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3993.371268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3993.980804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3994.622603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3995.287041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3995.904616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3996.545107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3997.131093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3997.760060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3998.403878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3999.058703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3999.676680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4000.325375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4000.936399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4001.495834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4002.088550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4002.730293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4003.292079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4003.920841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4004.494027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4005.150636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4005.780169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4006.469606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4007.128704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4007.838690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4008.511589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4009.184230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4009.888050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4010.520663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4011.086987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4011.702874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4012.282322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4012.842142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4013.465400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4014.073498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4014.689326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4015.326843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4015.965100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4016.565513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4017.242419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4017.913209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4018.593433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4019.218561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4019.780988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4020.403841] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4020.407097] Lustre: Skipped 207 previous similar messages [ 4020.418182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4020.424227] Lustre: Skipped 207 previous similar messages [ 4020.426781] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4020.431244] Lustre: Skipped 207 previous similar messages [ 4020.433651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4020.441238] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4020.444152] Lustre: Skipped 207 previous similar messages [ 4021.111336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4021.829795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4022.423632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4023.068527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4023.681845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4024.282924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4024.768278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4025.260779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4025.770200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4026.290486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4026.801467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4027.296523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4027.797772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4028.247957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4028.781075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4029.265358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4029.734465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4030.178862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4030.637981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4031.174192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4031.694222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4032.203292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4032.681123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4033.141888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4033.644850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4034.145635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4034.679092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4035.217991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4035.795460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4036.381454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4036.959539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4037.546915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4038.108102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4038.646769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4039.138676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4039.635695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4040.165639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4040.676057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4041.200626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4041.734815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4042.241561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4042.824422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4043.298740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4043.793792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4044.301607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4044.867608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4045.424700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4045.947081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4046.472756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4047.005592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4047.533533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4048.138170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4048.711338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4049.363661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4049.975262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4050.573433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4051.157889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4051.713756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4052.380416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4052.973514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4053.514949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4054.168918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4054.754545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4055.325501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4055.890420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4056.394871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4056.949943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4057.421097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4057.943575] LustreError: 162730:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4057.949074] LustreError: 162730:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4057.956483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4058.556139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4059.075605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4059.615293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4060.180640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4060.748307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4061.356113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4061.964893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4062.533637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4063.118060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4063.642614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4064.273703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4064.993917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4065.615503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4066.203893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4066.792368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4067.492057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4068.063200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4068.634408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4069.215091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4069.775777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4070.302556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4070.845986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4071.390685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4071.960986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4072.527443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4073.144805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4073.700667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4074.254810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4074.850557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4075.397245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4075.942034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4076.539260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4077.142355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4077.695507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4078.328905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4078.866665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4079.388985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4079.990781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4080.587380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4081.139916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4081.671627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4082.213069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4082.723830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4083.258800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4083.763250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4084.232961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4084.689631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4085.207380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4085.706515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4086.248815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4086.816808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4087.335689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4087.900547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4088.454116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4088.988082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4089.594560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4090.160148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4090.656805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4091.154882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4091.674307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4092.156911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4092.607078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4093.114416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4093.578524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4094.106344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4094.616255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4095.138590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4095.649978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4096.222913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4096.744729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4097.269245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4097.758581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4098.333817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4098.880865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4099.415129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4099.983426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4100.571528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4101.096997] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4101.737594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4102.306817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4102.873391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4103.398191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4103.903414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4104.438266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4105.012115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4105.463921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4105.991907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4106.598949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4107.114561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4107.694775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4108.194409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4108.704926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4109.190848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4109.681571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4110.176563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4110.669804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4111.159520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4111.694798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4112.154269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4112.658887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4113.076527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4113.565306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4114.075317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4114.519585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4114.967191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4115.439210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4115.861932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4116.374996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4116.902714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4117.434720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4117.924586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4118.418101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4118.923037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4119.456378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4119.964453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4120.435590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4120.969856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4121.489547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4121.941345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4122.468191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4123.053833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4123.551668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4124.109466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4124.624249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4125.084044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4125.583312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4126.124512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4126.602288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4127.093363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4127.614648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4128.118329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4128.555324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4129.090542] LustreError: 162730:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4129.096521] LustreError: 162730:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4129.101498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4129.595533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4130.140735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4130.634601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4131.190432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4131.744357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4132.277992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4132.788498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4133.285211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4133.743259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4134.237570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4134.824668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4135.349707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4135.860363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4136.359713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4136.831665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4137.327224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4137.746074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4138.166044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4138.629109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4139.054059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4139.495490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4140.054709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4140.567808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4141.013260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4141.501532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4141.984655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4142.416074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4142.850514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4143.273822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4143.729073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4144.179213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4144.566736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4145.017962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4145.530937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4146.004750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4146.461851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4146.966265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4147.550209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4148.146102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4148.714516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4149.248339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4149.745263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4150.290172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4150.799077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4151.243694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4151.728428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4152.241460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4152.762772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4153.262484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4153.780882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4154.291683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4154.822055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4155.362873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4155.870329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4156.348206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4156.839226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4157.296048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4157.787751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4158.288893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4158.850927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4159.380880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4159.871859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4160.453775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4161.009350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4161.526466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4162.046730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4162.566664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4163.011154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4163.522245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4164.061831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4164.698212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4165.172344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4165.722725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4166.247973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4166.798616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4167.345967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4167.913486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4168.438394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4168.955961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4169.444919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4170.018455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4170.536473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4171.059847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4171.540415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4172.096770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4172.577100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4173.148436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4173.617871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4174.102226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4174.638084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4175.178626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4175.713751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4176.205130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4176.686419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4177.252341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4177.764730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4178.264551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4178.799152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4179.381135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4179.965129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4180.556117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4181.165301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4181.681938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4182.180504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4182.710112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4183.190994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4183.670212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4184.164464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4184.652796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4185.220986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4185.749041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4186.351833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4186.930424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4187.496588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4188.076216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4188.571809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4189.058113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4189.571779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4190.016888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4190.484542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4190.959635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4191.461365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4191.929365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4192.374032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4192.925422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4193.378105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4193.829312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4194.301517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4194.774396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4195.278228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4195.845725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4196.418687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4196.897384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4197.421365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4197.935887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4198.493742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4199.011082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4199.616552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4200.109225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4200.601144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4201.056995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4201.560069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4201.985000] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4202.423439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4202.860017] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4203.293575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4203.746517] LustreError: 162730:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4203.751295] LustreError: 162730:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4203.752475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4204.178836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4204.575273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4205.010469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4205.422396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4205.838799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4206.287334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4206.715869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4207.134498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4207.544571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4207.975693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4208.361274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4208.776953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4209.236732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4209.671600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4210.123579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4210.583720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4211.072839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4211.553269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4212.038807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4212.484470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4212.924439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4213.442904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4213.922078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4214.379618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4214.818551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4215.300564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4215.749166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4216.260496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4216.844467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4217.371773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4217.909862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4218.380421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4218.866436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4219.355749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4219.865660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4220.376041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4220.850226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4221.299606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4221.737449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4222.211765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4222.730840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4223.300856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4223.907944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4224.556496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4225.170642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4225.717124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4226.271070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4226.810114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4227.318261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4227.830672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4228.399427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4228.907511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4229.442291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4229.899559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4230.381204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4230.864647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4231.322969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4231.865928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4232.364852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4232.806267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4233.274609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4233.831252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4234.331906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4234.784662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4235.233744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4235.696080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4236.136681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4236.655294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4237.176129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4237.711250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4238.208728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4238.691875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4239.180725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4239.644851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4240.144418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4240.596118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4241.063268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4241.566802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4242.077211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4242.572548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4243.018585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4243.538170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4244.005518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4244.461777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4244.893712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4245.329652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4245.761692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4246.188466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4246.591525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4247.045380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4247.480848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4247.894892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4248.310927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4248.717618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4249.159948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4249.603340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4250.033374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4250.512785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4251.004987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4251.492688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4251.961901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4252.461881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4252.964073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4253.395827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4253.884440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4254.420138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4254.865983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4255.278582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4255.833033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4256.309396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4256.757172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4257.238094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4257.760862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4258.261593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4258.702746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4259.209252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4259.641846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4260.089183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4260.549083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4260.998974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4261.461528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4261.972599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4262.517981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4262.985771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4263.526476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4263.943372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4264.417412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4264.881995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4265.337049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4265.797104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4266.233975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4266.693042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4267.227500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4267.797115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4268.298354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4268.752530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4269.247073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4269.709856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4270.157612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4270.590795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4271.055380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4271.523989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4272.065272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4272.600646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4273.185468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4273.689082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4274.189643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4274.671960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4275.072213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4275.493058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4275.890205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4276.297875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4276.705990] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4276.708681] Lustre: Skipped 501 previous similar messages [ 4276.714620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4276.719452] Lustre: Skipped 501 previous similar messages [ 4276.721375] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4276.724367] Lustre: Skipped 501 previous similar messages [ 4276.726616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4276.730503] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4276.732867] Lustre: Skipped 501 previous similar messages [ 4277.189067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4277.653927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4278.134124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4278.614878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4279.047285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4279.496662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4279.949401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4280.395375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4280.835900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4281.267123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4281.676338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4282.098354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4282.588145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4283.026393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4283.563551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4284.043924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4284.528173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4285.001724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4285.490369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4286.065896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4286.600453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4287.146804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4287.693535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4288.194891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4288.740179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4289.214859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4289.725305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4290.225399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4290.735541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4291.278714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4291.796976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4292.353258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4292.839118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4293.408612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4294.005334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4294.578449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4295.104875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4295.639651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4296.188937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4589.951357] LustreError: 164309:0:(ofd_dev.c:1776:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 4589.956663] LustreError: 162730:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 4589.960908] LustreError: 162730:0:(osp_precreate.c:651:osp_precreate_send()) Skipped 1 previous similar message [ 4765.170323] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4765.174500] LustreError: Skipped 4 previous similar messages [ 4765.174548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4765.910735] LustreError: 216532:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4766.107784] Lustre: server umount lustre-OST0000 complete [ 4769.761207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4769.764195] LustreError: Skipped 3 previous similar messages [ 4769.769195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4772.323335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4772.328044] Lustre: Skipped 1 previous similar message [ 4774.231337] LustreError: 216734:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4774.233735] LustreError: 216734:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4774.361235] Lustre: server umount lustre-MDT0000 complete [ 4776.223945] LustreError: 162691:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760130124 with bad export cookie 1488029447145385199 [ 4776.228698] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4776.230301] LustreError: 162691:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4785.638101] LustreError: 216936:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4785.641086] LustreError: 216936:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4785.770724] Lustre: server umount lustre-MDT0001 complete [ 4790.462514] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 4791.995955] Key type lgssc unregistered [ 4792.171791] LNet: 217466:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4792.175336] LNetError: 217466:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4792.188075] LNet: Removed LNI 192.168.204.112@tcp [ 4792.594063] Key type .llcrypt unregistered [ 4792.595977] Key type ._llcrypt unregistered [ 4805.525347] Key type ._llcrypt registered [ 4805.527266] Key type .llcrypt registered [ 4805.583389] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 4806.008383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4806.053100] alg: No test for adler32 (adler32-zlib) [ 4806.949329] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 4807.058948] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 4808.671320] Key type lgssc registered [ 4809.177098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4813.223647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4813.236439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4814.453126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4816.316875] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4820.565887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4820.742540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4822.510898] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4825.333247] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4827.199072] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4830.635721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4830.815204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4833.420933] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4836.432939] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4840.047110] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49666 to 0x280000401:49697) [ 4844.750388] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4845.597078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4845.603311] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4845.607711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4845.612768] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4845.612775] LustreError: 218858:0:(osp_precreate.c:1263:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 4846.562929] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4846.565378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4846.572041] Lustre: Skipped 1 previous similar message [ 4847.851934] Lustre: lustre-OST0000: Not available for connect from 192.168.204.12@tcp (stopping) [ 4847.855568] Lustre: Skipped 1 previous similar message [ 4851.681871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4851.684246] Lustre: Skipped 1 previous similar message [ 4852.529231] LustreError: 221237:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4852.614799] Lustre: server umount lustre-OST0000 complete [ 4865.695148] LustreError: 218858:0:(osp_precreate.c:1263:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 4881.680250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4881.869878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4884.450828] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4887.120123] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4889.040758] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 4890.977322] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8f8b10575000.ost_server_uuid 50 [ 4891.776915] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8f8b10575000.ost_server_uuid in FULL state after 0 sec [ 4899.561829] LustreError: 222552:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4899.565021] LustreError: 222552:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4899.655153] Lustre: server umount lustre-OST0000 complete [ 4901.476401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4901.484296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4901.486949] Lustre: Skipped 1 previous similar message [ 4907.614408] LustreError: 222753:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4907.616629] LustreError: 222753:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4907.675372] Lustre: server umount lustre-MDT0000 complete [ 4908.000290] LustreError: 219951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4908.007731] LustreError: 219951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4909.348244] LustreError: 220456:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760130258 with bad export cookie 15842728388875589636 [ 4909.351722] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4909.353326] LustreError: 220456:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4909.525932] Lustre: server umount lustre-MDT0001 complete [ 4914.523946] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 4915.785273] Key type lgssc unregistered [ 4915.939628] LNet: 223485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4915.943494] LNetError: 223485:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4915.953744] LNet: Removed LNI 192.168.204.112@tcp [ 4916.313277] Key type .llcrypt unregistered [ 4916.314776] Key type ._llcrypt unregistered [ 4931.887595] Key type ._llcrypt registered [ 4931.889153] Key type .llcrypt registered [ 4931.942383] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 4932.324920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4932.333051] alg: No test for adler32 (adler32-zlib) [ 4933.218060] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 4933.316651] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 4934.911241] Key type lgssc registered [ 4935.392855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4939.394144] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4939.403686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4940.624468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4942.352330] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4945.875084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4946.041719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4947.684132] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4948.891530] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4951.628490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4951.795959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4954.137720] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4957.713751] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4957.753351] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4957.825854] Lustre: lustre-OST0001: new disk, initializing [ 4957.827690] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4957.870942] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4960.211232] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4964.359877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4964.363901] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4964.397655] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4965.418675] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49699 to 0x280000401:49729) [ 4970.421225] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4974.473356] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4990.949641] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4990.951180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4990.954018] Lustre: Skipped 2 previous similar messages [ 4990.956271] Lustre: Skipped 1 previous similar message [ 4993.863177] LustreError: 228535:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4993.910736] Lustre: server umount lustre-MDT0000 complete [ 4995.368724] LustreError: 227436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760130344 with bad export cookie 17884697358176191098 [ 4995.372195] LustreError: 225893:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200001b72:0x1:0x0]: rc = -5 [ 4995.372503] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4995.373658] LustreError: 227436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4995.437240] LustreError: 228736:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4995.439641] LustreError: 228736:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4995.546843] Lustre: server umount lustre-MDT0001 complete [ 5007.281086] LustreError: 228937:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5007.282939] LustreError: 228937:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5007.299858] Lustre: server umount lustre-OST0000 complete [ 5018.867126] LustreError: 229139:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5018.869146] LustreError: 229139:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5018.949362] Lustre: server umount lustre-OST0001 complete [ 5020.169393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5020.218141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5020.656676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5020.701111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5021.200429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5021.233491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5021.685733] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5021.711902] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5027.649815] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 5031.747603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5031.926831] LustreError: 230546:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5031.936029] LustreError: 230546:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5031.954689] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5033.428536] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5036.645201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5038.219543] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5039.340128] Lustre: 231656:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5041.921268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5042.085090] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5042.088442] Lustre: Skipped 1 previous similar message [ 5044.358825] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5047.208533] LustreError: 232015:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5047.217331] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:100 to 0x280000400:129) [ 5047.897962] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5050.383662] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5056.367280] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49699 to 0x280000401:49761) [ 5059.504240] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5061.026063] Lustre: 233504:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5071.843436] Lustre: mylustre-MDT0000-lwp-OST0000: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5071.846938] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5071.848882] Lustre: Skipped 3 previous similar messages [ 5071.851644] Lustre: Skipped 1 previous similar message [ 5073.093281] LustreError: 233810:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5073.095974] LustreError: 233810:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5073.127453] Lustre: server umount mylustre-MDT0000 complete [ 5074.577120] LustreError: 232036:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760130423 with bad export cookie 17884697358176193954 [ 5074.579555] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5074.581580] LustreError: 232036:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5074.775492] Lustre: server umount mylustre-MDT0001 complete [ 5087.157213] LustreError: 234215:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 5087.160512] LustreError: 234215:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 5087.182231] Lustre: server umount mylustre-OST0000 complete [ 5098.950283] Lustre: server umount mylustre-OST0001 complete [ 5100.217287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5100.270530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5100.724668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5100.762308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5101.217237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5101.251943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5101.698071] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5101.741612] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5107.548657] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 5111.789028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5112.017189] LustreError: 235826:0:(ldlm_lib.c:1179:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5112.056865] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5112.059816] Lustre: Skipped 1 previous similar message [ 5113.716387] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5117.162108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5119.053482] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5120.261059] Lustre: 236937:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5122.921509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5125.413877] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5128.225700] LustreError: 237992:0:(ldlm_lib.c:1179:target_handle_connect()) tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5128.229210] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 5128.949852] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5131.154161] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5136.431964] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49699 to 0x280000401:49793) [ 5139.796784] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5141.186621] Lustre: 238785:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5151.716821] Lustre: tfs-MDT0000-lwp-OST0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5151.719713] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 5151.721820] Lustre: Skipped 4 previous similar messages [ 5151.724544] Lustre: Skipped 3 previous similar messages [ 5154.692939] LustreError: 239035:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5154.695683] LustreError: 239035:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5154.734856] Lustre: server umount tfs-MDT0000 complete [ 5156.124398] LustreError: 235802:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760130504 with bad export cookie 17884697358176195907 [ 5156.128172] LustreError: 235802:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5156.128206] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5156.284841] Lustre: server umount tfs-MDT0001 complete [ 5167.942178] Lustre: server umount tfs-OST0000 complete [ 5178.562968] Lustre: server umount tfs-OST0001 complete [ 5179.685610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5179.718718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5180.114046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5180.142331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5180.557527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5180.590090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5180.984598] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5181.016336] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5186.227479] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 5190.105694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5190.288978] LustreError: 241046:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5190.316174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5190.319220] Lustre: Skipped 3 previous similar messages [ 5191.727343] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5194.738663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5196.271610] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5197.372923] Lustre: 242156:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5199.900353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5201.059057] LustreError: 242512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5201.063040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49699 to 0x280000401:49825) [ 5202.297346] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5205.161126] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 5205.447342] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5207.606340] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5210.802590] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5212.071445] Lustre: 244003:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5219.416706] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 17:09:28 (1760130568) [ 5236.197274] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5236.198487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5236.200128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5236.200139] Lustre: Skipped 1 previous similar message [ 5236.200629] Lustre: Skipped 2 previous similar messages [ 5237.438281] LustreError: 244372:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5237.441038] LustreError: 244372:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 5237.474778] Lustre: server umount lustre-MDT0000 complete [ 5238.719168] LustreError: 241020:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760130587 with bad export cookie 17884697358176197601 [ 5238.721904] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5238.723266] LustreError: 241020:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5241.312480] LustreError: 241040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5241.313234] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5241.314650] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5241.314658] Lustre: Skipped 3 previous similar messages [ 5241.318483] LustreError: 241040:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5241.321932] Lustre: Skipped 1 previous similar message [ 5253.087165] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5253.209556] Lustre: server umount lustre-MDT0001 complete [ 5260.670869] Lustre: server umount lustre-OST0000 complete [ 5268.157661] Lustre: server umount lustre-OST0001 complete [ 5270.507815] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 5273.849730] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 5277.877272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5280.270310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5282.471312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5284.794084] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5287.268159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5287.287378] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 5287.287378] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 5287.287378] [ 5287.294755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 5287.385966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5287.402121] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5287.441904] Lustre: lustre-MDT0000: new disk, initializing [ 5287.476547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5287.479797] Lustre: Skipped 3 previous similar messages [ 5287.487601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5289.062282] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5293.526764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5293.553067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5293.590660] Lustre: 247158:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5293.617197] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5293.618876] Lustre: Skipped 1 previous similar message [ 5293.659639] Lustre: lustre-MDT0001: new disk, initializing [ 5293.709529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5293.713115] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5295.110500] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5299.399723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5299.424599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5299.547050] Lustre: lustre-OST0000: new disk, initializing [ 5299.549701] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5301.433326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5301.436636] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5301.474530] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5301.650082] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5305.886938] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5305.915580] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5305.973444] Lustre: lustre-OST0001: new disk, initializing [ 5305.975099] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5307.013827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5307.017973] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5307.054422] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5308.014990] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5312.294481] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 17:11:01 (1760130661) [ 5315.922056] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 17:11:04 (1760130664) [ 5332.448357] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5332.448405] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5332.451758] LustreError: Skipped 1 previous similar message [ 5332.456260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5332.459380] Lustre: Skipped 6 previous similar messages [ 5333.421116] LustreError: 250582:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5333.423701] LustreError: 250582:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 5333.445703] Lustre: server umount lustre-OST0000 complete [ 5335.008049] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5338.082383] LustreError: 248696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5338.086195] LustreError: 248696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5349.343181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5349.404702] Lustre: server umount lustre-MDT0000 complete [ 5350.644108] LustreError: 247149:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760130699 with bad export cookie 17884697358176199155 [ 5350.647390] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5350.648028] LustreError: 247149:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5353.440202] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5353.443377] Lustre: Skipped 4 previous similar messages [ 5353.444751] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5353.446434] Lustre: Skipped 10 previous similar messages [ 5356.889858] Lustre: server umount lustre-MDT0001 complete [ 5360.660753] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 5370.628904] Lustre: server umount lustre-OST0001 complete [ 5372.935439] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 5379.197037] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 5382.818233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5385.186638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5387.468622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5389.837546] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5392.314941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5392.341842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5392.452946] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5392.470748] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5392.513605] Lustre: lustre-MDT0000: new disk, initializing [ 5392.562482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5394.147902] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5398.763746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5398.793529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5398.842531] Lustre: 254043:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5398.978187] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5400.564480] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5403.966787] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5405.266468] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5407.713565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5407.737187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5409.154840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5409.159255] Lustre: Skipped 1 previous similar message [ 5409.162090] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5409.198902] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5410.177513] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5413.567435] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5419.490667] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5419.490791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5419.491806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5419.491811] Lustre: Skipped 1 previous similar message [ 5419.503154] Lustre: Skipped 1 previous similar message [ 5422.197383] Lustre: server umount lustre-OST0000 complete [ 5424.610847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5424.613725] LustreError: Skipped 1 previous similar message [ 5429.719618] Lustre: server umount lustre-MDT0000 complete [ 5429.728069] LustreError: 254053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5429.733189] LustreError: 254053:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5431.122138] LustreError: 254825:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760130779 with bad export cookie 17884697358176200625 [ 5431.125508] LustreError: 254825:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5431.125564] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5431.288689] Lustre: server umount lustre-MDT0001 complete [ 5435.238194] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 5436.485424] Key type lgssc unregistered [ 5436.636449] LNet: 257421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5436.639654] LNetError: 257421:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5436.652297] LNet: Removed LNI 192.168.204.112@tcp [ 5437.034178] Key type .llcrypt unregistered [ 5437.035761] Key type ._llcrypt unregistered [ 5447.708966] Key type ._llcrypt registered [ 5447.710686] Key type .llcrypt registered [ 5447.761865] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 5448.153218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5448.225138] alg: No test for adler32 (adler32-zlib) [ 5449.094857] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 5449.195236] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 5450.783229] Key type lgssc registered [ 5451.197090] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5454.030688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5454.038846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5455.212573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5456.553542] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5459.463912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5459.597176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5460.872701] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5462.692196] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5463.952906] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5466.334381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5466.468130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5468.541097] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5470.620767] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5470.975709] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 5474.787955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5474.788440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5474.795799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5477.676077] LustreError: 261135:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5477.743672] Lustre: server umount lustre-OST0000 complete [ 5479.904645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5479.905466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5479.908858] Lustre: Skipped 1 previous similar message [ 5485.024462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5485.027363] Lustre: Skipped 2 previous similar messages [ 5485.120661] LustreError: 261340:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5485.123067] LustreError: 261340:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5485.157950] Lustre: server umount lustre-MDT0000 complete [ 5486.407321] LustreError: 258821:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760130835 with bad export cookie 8498686125539761494 [ 5486.411957] LustreError: 258821:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5486.412248] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5486.471985] LustreError: 261541:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5486.473495] LustreError: 261541:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5486.554924] Lustre: server umount lustre-MDT0001 complete [ 5490.331616] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 5491.426437] Key type lgssc unregistered [ 5491.574437] LNet: 262072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5491.577661] LNetError: 262072:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5491.588415] LNet: Removed LNI 192.168.204.112@tcp [ 5491.921487] Key type .llcrypt unregistered [ 5491.922397] Key type ._llcrypt unregistered [ 5498.565882] Key type ._llcrypt registered [ 5498.566828] Key type .llcrypt registered [ 5498.605233] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 5498.921167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5498.940635] alg: No test for adler32 (adler32-zlib) [ 5499.814742] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 5499.905131] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 5501.495151] Key type lgssc registered [ 5501.864966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5505.079079] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5505.086601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5520.543337] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5525.663211] LustreError: 263221:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.112@tcp: failed processing log, type 4: rc = -110 [ 5552.287223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5554.401739] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5556.426205] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5557.223346] LustreError: 263886:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 5557.294063] Lustre: server umount lustre-OST0000 complete [ 5563.132968] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 5564.160262] Key type lgssc unregistered [ 5564.295451] LNet: 264721:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5564.297557] LNetError: 264721:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5564.305475] LNet: Removed LNI 192.168.204.112@tcp [ 5564.598207] Key type .llcrypt unregistered [ 5564.599132] Key type ._llcrypt unregistered [ 5579.231356] Key type ._llcrypt registered [ 5579.232658] Key type .llcrypt registered [ 5579.276377] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 5579.701628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5579.766194] alg: No test for adler32 (adler32-zlib) [ 5580.642080] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 5580.741853] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 5582.335162] Key type lgssc registered [ 5582.776041] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5587.468356] loop0: detected capacity change from 0 to 204800000 [ 5587.570798] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5587.574804] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5587.720312] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5587.723872] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5587.744303] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5587.767870] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5587.787134] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5587.817901] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5587.852419] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5587.889558] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5588.273594] loop0: detected capacity change from 0 to 204800000 [ 5588.334966] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5588.603102] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5588.960584] loop0: detected capacity change from 0 to 204800000 [ 5589.150540] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5589.527339] loop0: detected capacity change from 0 to 204800000 [ 5589.684284] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5590.027485] loop0: detected capacity change from 0 to 204800000 [ 5590.037058] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 5591.440265] loop0: detected capacity change from 0 to 204800000 [ 5591.454428] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 5592.880179] loop0: detected capacity change from 0 to 204800000 [ 5592.888402] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 5594.338765] loop0: detected capacity change from 0 to 204800000 [ 5594.349117] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 5595.744841] loop0: detected capacity change from 0 to 204800000 [ 5595.784384] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5595.815603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5595.822105] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5595.847161] Lustre: 268094:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 5595.852204] Lustre: 268094:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 5595.856235] Lustre: 268094:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 5595.860845] Lustre: 268094:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 5595.864667] Lustre: 268094:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 5595.868276] Lustre: 268094:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 5595.871994] Lustre: 268094:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 5595.876375] Lustre: 268094:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 5595.879307] Lustre: 268094:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 5595.883574] Lustre: 268094:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 5595.887074] Lustre: 268094:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 5595.890756] Lustre: 268094:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 5595.894866] Lustre: 268094:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 5595.899563] Lustre: 268094:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 5598.655656] LustreError: 268360:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5598.704044] Lustre: server umount MGS complete [ 5599.023465] loop0: detected capacity change from 0 to 204800000 [ 5599.067510] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5599.254924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5599.275238] Lustre: lustre-MDT0000: Aborting client recovery [ 5599.606345] loop1: detected capacity change from 0 to 204800000 [ 5599.644612] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5599.675865] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5599.697813] Lustre: 268532:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 5599.701593] Lustre: 268532:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 5599.705283] Lustre: 268532:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 5599.709324] Lustre: 268532:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 5599.712284] Lustre: 268532:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 5599.715532] Lustre: 268532:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 5599.718530] Lustre: 268532:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 5599.725309] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5599.799520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5599.828714] Lustre: lustre-MDT0001: Aborting client recovery [ 5600.147783] loop2: detected capacity change from 0 to 204800000 [ 5600.185628] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5600.215969] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5600.235644] Lustre: 268616:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 5600.241563] Lustre: 268616:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 5600.247497] Lustre: 268616:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 5600.253412] Lustre: 268616:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 5600.259387] Lustre: 268616:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 5600.264079] Lustre: 268616:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 5600.268612] Lustre: 268616:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 5600.323296] Lustre: Found index 0 for lustre-OST0000, updating log [ 5600.358681] Lustre: lustre-OST0000: Aborting recovery [ 5600.665855] loop3: detected capacity change from 0 to 204800000 [ 5600.728942] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5600.748531] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5600.772280] Lustre: 268703:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 5600.777734] Lustre: 268703:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 5600.783510] Lustre: 268703:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 5600.787055] Lustre: 268703:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 5600.791524] Lustre: 268703:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 5600.795209] Lustre: 268703:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 5600.799207] Lustre: 268703:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 5600.836482] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5600.839017] Lustre: Skipped 1 previous similar message [ 5608.993837] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 5608.994628] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 5609.005920] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 5609.007368] LustreError: 268475:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 5609.009519] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 5610.018346] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 5610.018390] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 5611.214151] LustreError: 268440:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 254us: rc = -2 [ 5625.312816] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5625.313263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5625.317255] Lustre: Skipped 3 previous similar messages [ 5625.319492] Lustre: Skipped 2 previous similar messages [ 5625.322337] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5628.473221] LustreError: 269060:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5628.475690] LustreError: 269060:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5628.510819] Lustre: server umount lustre-MDT0000 complete [ 5628.836989] LustreError: 269010:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760130977 with bad export cookie 17816747795949070546 [ 5628.838977] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5628.841304] LustreError: 269010:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5630.431850] LustreError: 268436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5630.432180] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5630.432478] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5630.432482] Lustre: Skipped 1 previous similar message [ 5630.437335] LustreError: 268436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5630.442935] Lustre: Skipped 1 previous similar message [ 5633.439511] LustreError: 268436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5633.439579] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5633.444638] LustreError: 268436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5633.447984] Lustre: Skipped 2 previous similar messages [ 5638.559618] LustreError: 268937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5638.559703] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5638.566657] LustreError: 268937:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5638.568190] Lustre: Skipped 1 previous similar message [ 5643.231151] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5643.265015] LustreError: 269111:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 5643.267596] LustreError: 269111:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5643.358935] Lustre: server umount lustre-MDT0001 complete [ 5649.770081] LustreError: 269163:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 5649.773589] LustreError: 269163:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5649.796972] Lustre: server umount lustre-OST0000 complete [ 5656.171864] LustreError: 269214:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5656.175246] LustreError: 269214:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5656.253452] Lustre: server umount lustre-OST0001 complete [ 5658.915322] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 17:16:47 (1760131007) [ 5667.840834] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 5670.761973] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 5674.347856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5676.477991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5678.424248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5680.525669] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5682.745941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5682.767191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5682.846130] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5682.855614] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5682.886229] Lustre: lustre-MDT0000: new disk, initializing [ 5682.907962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5682.914396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5684.185837] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5688.312661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5688.333760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5688.366173] Lustre: 272755:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5688.394956] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5688.397071] Lustre: Skipped 1 previous similar message [ 5688.431276] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5688.445739] Lustre: lustre-MDT0001: new disk, initializing [ 5688.480901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5688.495407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5688.499404] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5689.834178] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5693.121717] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5694.351797] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5696.725904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5696.750731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5696.855496] Lustre: lustre-OST0000: new disk, initializing [ 5696.857515] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5696.891103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5698.869768] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5701.929813] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5703.157772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5703.160755] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5703.193850] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5709.741371] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 5719.898788] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 5725.663464] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 5769.697961] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5769.698635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5769.699191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5769.700317] LustreError: Skipped 1 previous similar message [ 5769.704820] Lustre: Skipped 1 previous similar message [ 5772.974112] LustreError: 276516:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5772.976864] LustreError: 276516:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5773.044446] Lustre: server umount lustre-OST0000 complete [ 5774.816357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5774.819883] Lustre: Skipped 1 previous similar message [ 5780.436279] Lustre: server umount lustre-MDT0000 complete [ 5781.717279] LustreError: 272747:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760131130 with bad export cookie 17816747795949073248 [ 5781.722065] LustreError: 272747:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5781.722251] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5781.870689] Lustre: server umount lustre-MDT0001 complete [ 5784.799096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5786.067033] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5787.761567] Lustre: server umount MGS complete [ 5791.454509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5791.598173] LustreError: 278482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5791.604541] LustreError: 278494:0:(qmt_pool.c:1513:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 5791.630524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5793.005888] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5795.866834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5797.360772] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5799.222647] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5800.454959] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5802.828995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5804.823558] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5806.635036] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5842.912887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5842.914074] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5842.914489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5842.914495] Lustre: Skipped 5 previous similar messages [ 5842.918382] Lustre: Skipped 1 previous similar message [ 5848.941184] LustreError: 281164:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5848.943415] LustreError: 281164:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 5849.006835] Lustre: server umount lustre-OST0000 complete [ 5852.639837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5852.641853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5856.407168] Lustre: server umount lustre-MDT0000 complete [ 5857.597215] LustreError: 278462:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760131206 with bad export cookie 17816747795949075075 [ 5857.600685] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5857.601723] LustreError: 278462:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5857.775450] Lustre: server umount lustre-MDT0001 complete [ 5861.365692] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 5862.458814] Key type lgssc unregistered [ 5862.587433] LNet: 282096:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5862.590496] LNetError: 282096:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5862.600716] LNet: Removed LNI 192.168.204.112@tcp [ 5862.935156] Key type .llcrypt unregistered [ 5862.936235] Key type ._llcrypt unregistered [ 5874.763866] Key type ._llcrypt registered [ 5874.764910] Key type .llcrypt registered [ 5874.803606] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 5880.761024] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 5881.185991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5881.242410] alg: No test for adler32 (adler32-zlib) [ 5882.105655] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 5882.181615] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 5883.759155] Key type lgssc registered [ 5884.125541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5887.017341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5889.140098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5891.202217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5893.463136] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5895.971664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5895.992642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5895.998549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5897.090931] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5897.103177] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5897.141139] Lustre: lustre-MDT0000: new disk, initializing [ 5897.171127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5897.181153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5898.668511] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5903.116167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5903.138618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5903.173387] Lustre: 285999:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5903.196173] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5903.197860] Lustre: Skipped 1 previous similar message [ 5903.230883] Lustre: lustre-MDT0001: new disk, initializing [ 5903.263504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5903.274182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5903.276976] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5904.599453] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5907.730760] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5908.888474] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5911.059650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5911.081485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5911.174230] Lustre: lustre-OST0000: new disk, initializing [ 5911.175877] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5911.205184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5913.033260] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5916.071457] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5917.687984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5917.690905] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5917.717938] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5924.705236] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 5929.628540] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 5938.072530] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 5948.882605] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 5994.465071] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5994.467392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5994.468529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5994.469257] LustreError: Skipped 1 previous similar message [ 5994.472402] Lustre: Skipped 1 previous similar message [ 5999.588474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5999.591020] Lustre: Skipped 2 previous similar messages [ 6004.708312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6004.711370] Lustre: Skipped 1 previous similar message [ 6005.215171] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6005.233138] LustreError: 289687:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6005.300131] Lustre: server umount lustre-OST0000 complete [ 6009.825698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6009.826792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6009.831417] Lustre: Skipped 1 previous similar message [ 6012.734357] LustreError: 289889:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6012.736067] LustreError: 289889:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6012.768781] Lustre: server umount lustre-MDT0000 complete [ 6014.102775] LustreError: 285990:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760131362 with bad export cookie 12806954508150754752 [ 6014.106511] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6014.108236] LustreError: 285990:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6014.175118] LustreError: 290090:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6014.178232] LustreError: 290090:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6014.273321] Lustre: server umount lustre-MDT0001 complete [ 6017.263186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6018.601313] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6020.308674] LustreError: 291031:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6020.311702] LustreError: 291031:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6020.336733] Lustre: server umount MGS complete [ 6024.207648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6024.362574] LustreError: 291667:0:(qmt_pool.c:1513:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 6024.386452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6025.790161] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6028.969385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6029.125710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6030.489186] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6032.236433] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6033.388552] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6035.566687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6037.417521] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6039.126700] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6074.847873] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6074.850424] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6074.855391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6074.857172] Lustre: Skipped 1 previous similar message [ 6076.896846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6076.898775] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6078.575177] LustreError: 294337:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6078.577092] LustreError: 294337:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6078.653462] Lustre: server umount lustre-OST0000 complete [ 6080.479892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6082.017112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6082.021306] Lustre: Skipped 1 previous similar message [ 6087.136218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6087.138760] Lustre: Skipped 6 previous similar messages [ 6094.303148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6094.335900] LustreError: 294538:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6094.337612] LustreError: 294538:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6094.362359] Lustre: server umount lustre-MDT0000 complete [ 6095.605569] LustreError: 293274:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760131444 with bad export cookie 12806954508150756649 [ 6095.609897] LustreError: 293274:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6095.610030] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6095.761737] Lustre: server umount lustre-MDT0001 complete [ 6099.331921] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 6100.443661] Key type lgssc unregistered [ 6100.588365] LNet: 295272:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6100.590728] LNetError: 295272:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6100.597314] LNet: Removed LNI 192.168.204.112@tcp [ 6100.908095] Key type .llcrypt unregistered [ 6100.909746] Key type ._llcrypt unregistered [ 6107.874125] Key type ._llcrypt registered [ 6107.875405] Key type .llcrypt registered [ 6107.915844] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 6113.710707] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 6114.200914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6114.230603] alg: No test for adler32 (adler32-zlib) [ 6115.095596] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 6115.193505] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 6116.783196] Key type lgssc registered [ 6117.203121] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6120.301166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6122.405330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6125.625963] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 6128.617480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6130.851657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6130.870244] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6130.875686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6131.961177] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6131.973329] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6132.008701] Lustre: lustre-MDT0000: new disk, initializing [ 6132.035291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6132.042116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6133.390307] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6137.522947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6137.550060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6137.648592] Lustre: lustre-OST0000: new disk, initializing [ 6137.650292] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6137.651654] Lustre: Skipped 1 previous similar message [ 6137.673752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6138.237910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6138.241376] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6138.249884] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6139.585268] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6322.657254] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6322.664301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6327.621706] LustreError: 300180:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6327.817750] Lustre: server umount lustre-MDT0000 complete [ 6329.225319] LustreError: 298434:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760131677 with bad export cookie 15791711761174706114 [ 6329.227472] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6329.230731] LustreError: 298434:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6329.258735] LustreError: 300385:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6329.261926] LustreError: 300385:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6329.338725] Lustre: server umount lustre-OST0000 complete [ 6337.933973] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 6341.005399] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 6344.224867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6346.336110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6348.265653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6350.269691] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6352.444884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6352.466324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6352.544480] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6352.554813] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6352.587889] Lustre: lustre-MDT0000: new disk, initializing [ 6352.612660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6352.619594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6353.870495] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6357.859302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6357.880900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6357.940791] Lustre: 303420:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6357.967841] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6357.970406] Lustre: Skipped 1 previous similar message [ 6358.007390] Lustre: lustre-MDT0001: new disk, initializing [ 6358.043476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6358.055792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6358.058923] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6359.307264] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6362.324042] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6363.499357] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6365.707466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6365.730125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6365.839526] Lustre: lustre-OST0000: new disk, initializing [ 6365.841631] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6365.869743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6367.417310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6367.421027] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6367.455543] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6367.712285] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6370.760770] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6372.111417] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6372.183710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6373.549173] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6373.626142] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6377.953189] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6377.954990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6377.956109] LustreError: Skipped 1 previous similar message [ 6377.956799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6377.960669] Lustre: Skipped 1 previous similar message [ 6380.332613] LustreError: 306358:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6380.334539] LustreError: 306358:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6380.394514] Lustre: server umount lustre-OST0000 complete [ 6383.072717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6383.073412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6383.076949] Lustre: Skipped 1 previous similar message [ 6383.078811] Lustre: Skipped 1 previous similar message [ 6387.705916] LustreError: 306559:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6387.707582] LustreError: 306559:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6387.732379] Lustre: server umount lustre-MDT0000 complete [ 6388.936207] LustreError: 303413:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760131737 with bad export cookie 15791711761177646891 [ 6388.938958] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6388.939714] LustreError: 303413:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6389.089499] Lustre: server umount lustre-MDT0001 complete [ 6390.191473] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [ 6390.727349] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 17:28:59 (1760131739) [ 6392.889575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6393.056343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6394.377687] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6397.184573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6398.479966] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6400.084288] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6401.186862] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6403.345970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6405.276237] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6406.951913] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6409.187463] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6409.212944] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6409.276333] Lustre: lustre-OST0001: new disk, initializing [ 6409.279327] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6409.320240] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6409.323138] Lustre: Skipped 2 previous similar messages [ 6411.161631] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6415.877076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6415.881490] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6415.901871] LustreError: 310390:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 6416.930165] LustreError: 310394:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 6416.934352] LustreError: 310394:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 6417.967856] LustreError: 310390:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 6417.972068] LustreError: 310390:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 6420.001464] LustreError: 310394:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 6420.005306] LustreError: 310394:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 6420.009025] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6420.093166] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6421.297118] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8f8b098c8800.ost_server_uuid 50 [ 6421.839489] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8f8b098c8800.ost_server_uuid in FULL state after 0 sec [ 6423.038888] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f8b098c8800.ost_server_uuid 50 [ 6423.523130] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8f8b098c8800.ost_server_uuid in FULL state after 0 sec [ 6424.101425] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6438.378414] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6438.380768] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6438.384981] Lustre: Skipped 1 previous similar message [ 6438.389317] Lustre: Skipped 2 previous similar messages [ 6441.198850] LustreError: 311138:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6441.202342] LustreError: 311138:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6441.227207] Lustre: server umount lustre-OST0001 complete [ 6443.490420] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6443.491445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6443.492043] LustreError: 309436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6443.492051] LustreError: 309436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6443.502931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6443.504855] Lustre: Skipped 2 previous similar messages [ 6448.614490] LustreError: 309436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6448.621524] LustreError: 309436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6449.076598] Lustre: server umount lustre-OST0000 complete [ 6453.729055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6453.729183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6453.730205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6453.730210] Lustre: Skipped 2 previous similar messages [ 6453.733676] Lustre: Skipped 2 previous similar messages [ 6456.445722] LustreError: 311540:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6456.447603] LustreError: 311540:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6456.472182] Lustre: server umount lustre-MDT0000 complete [ 6457.695217] LustreError: 307312:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760131806 with bad export cookie 15791711761177647759 [ 6457.698759] LustreError: 307312:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6457.698799] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6457.855153] Lustre: server umount lustre-MDT0001 complete [ 6461.555375] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 6462.701228] Key type lgssc unregistered [ 6462.839375] LNet: 312272:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6462.841591] LNetError: 312272:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6462.854302] LNet: Removed LNI 192.168.204.112@tcp [ 6463.178316] Key type .llcrypt unregistered [ 6463.180504] Key type ._llcrypt unregistered [ 6472.203641] Key type ._llcrypt registered [ 6472.205119] Key type .llcrypt registered [ 6472.244079] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 6472.524519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6472.534892] alg: No test for adler32 (adler32-zlib) [ 6473.391700] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 6473.478975] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 6475.063215] Key type lgssc registered [ 6475.409589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6477.895947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6477.902586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6479.101185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6480.408710] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6482.250179] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6484.597332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6484.742841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6486.181837] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6489.430626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6489.557530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6491.475177] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6493.242194] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6494.692819] LustreError: 315944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6494.703293] LustreError: 315944:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6495.602207] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6495.680274] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6497.561661] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6499.345326] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6504.233874] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6504.234166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 6507.037025] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6507.132493] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6586.344469] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6586.344891] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6586.346350] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6586.348951] Lustre: Skipped 1 previous similar message [ 6589.419141] LustreError: 317069:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6589.436738] Lustre: server umount lustre-OST0001 complete [ 6591.457121] LustreError: 315944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6591.458134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6591.459691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6591.459698] Lustre: Skipped 1 previous similar message [ 6591.460750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6591.464492] LustreError: 315944:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6591.478624] Lustre: Skipped 1 previous similar message [ 6596.577784] LustreError: 315198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6596.577804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6596.577816] Lustre: Skipped 1 previous similar message [ 6596.584630] LustreError: 315198:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6597.164110] LustreError: 317270:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6597.167265] LustreError: 317270:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6597.258544] Lustre: server umount lustre-OST0000 complete [ 6601.696698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6601.697758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6601.700339] Lustre: Skipped 1 previous similar message [ 6604.738257] LustreError: 317471:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6604.741434] LustreError: 317471:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6604.778384] Lustre: server umount lustre-MDT0000 complete [ 6606.074147] LustreError: 313630:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760131954 with bad export cookie 1126395368823776943 [ 6606.078517] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6606.079836] LustreError: 313630:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6606.244458] Lustre: server umount lustre-MDT0001 complete [ 6610.274333] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 6611.530160] Key type lgssc unregistered [ 6611.678777] LNet: 318204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6611.682378] LNetError: 318204:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6611.693450] LNet: Removed LNI 192.168.204.112@tcp [ 6612.038152] Key type .llcrypt unregistered [ 6612.039320] Key type ._llcrypt unregistered [ 6630.220244] Key type ._llcrypt registered [ 6630.221457] Key type .llcrypt registered [ 6630.265309] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 6630.598796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6630.635031] alg: No test for adler32 (adler32-zlib) [ 6631.485293] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 6631.565572] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 6633.143151] Key type lgssc registered [ 6633.515263] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6636.676433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6636.684986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6637.865700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6639.154384] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6642.046377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6642.175306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6643.442018] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6644.405600] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6646.651217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6646.779561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6647.781616] LustreError: 321910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6647.785855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 6647.786915] LustreError: 321910:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6648.668298] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6651.680544] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6651.768451] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6652.784589] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 6653.748748] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6657.924310] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6661.680339] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6672.864129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6672.868112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6672.874819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6673.379823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6673.383298] Lustre: Skipped 1 previous similar message [ 6673.384175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6673.386202] Lustre: Skipped 1 previous similar message [ 6676.480403] LustreError: 323863:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6676.507341] Lustre: server umount lustre-MDT0000 complete [ 6677.806101] LustreError: 320428:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760132026 with bad export cookie 18002641596697766464 [ 6677.809802] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6677.810778] LustreError: 320428:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6677.874561] LustreError: 324065:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 6677.877389] LustreError: 324065:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6677.972031] Lustre: server umount lustre-MDT0001 complete [ 6689.522110] LustreError: 324265:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6689.524562] LustreError: 324265:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6689.540356] Lustre: server umount lustre-OST0000 complete [ 6701.237784] LustreError: 324467:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6701.240422] LustreError: 324467:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6701.312938] Lustre: server umount lustre-OST0001 complete [ 6706.947104] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 6710.983054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6711.151637] LustreError: 325643:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6711.174736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6712.578611] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6715.672144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6717.259235] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6718.281588] Lustre: 326751:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6720.594650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6720.736898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6720.741093] Lustre: Skipped 1 previous similar message [ 6721.768788] LustreError: 327108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6721.773710] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 6722.730557] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6725.740214] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6726.890947] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 6727.809184] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6731.041291] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6732.276174] Lustre: 328600:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6741.471984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6741.475694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6741.481491] Lustre: Skipped 1 previous similar message [ 6741.484079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6741.486986] Lustre: Skipped 2 previous similar messages [ 6744.320554] LustreError: 328703:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6744.322380] LustreError: 328703:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6744.353360] Lustre: server umount lustre-MDT0000 complete [ 6745.614390] LustreError: 325622:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760132094 with bad export cookie 18002641596697768123 [ 6745.617701] LustreError: 325622:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6745.617929] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6745.786551] Lustre: server umount lustre-MDT0001 complete [ 6756.788106] LustreError: 329107:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6756.791713] LustreError: 329107:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 6756.809360] Lustre: server umount lustre-OST0000 complete [ 6768.313024] Lustre: server umount lustre-OST0001 complete [ 6769.537507] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [ 6770.085508] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [ 6770.669461] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 17:35:19 (1760132119) [ 6778.304180] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 6781.588198] loop0: detected capacity change from 0 to 1099511627776 [ 6781.772266] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 6781.776146] *** VALIDATE xfs *** [ 6781.781086] XFS (loop0): Mounting V5 Filesystem [ 6781.791462] XFS (loop0): Ending clean mount [ 6782.236635] loop1: detected capacity change from 0 to 18691697672192 [ 6799.036569] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6805.125625] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 6808.719475] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 6812.473356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6814.939661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6816.951425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6819.013094] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6821.278251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6821.301845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6821.385475] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6821.397954] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6821.434844] Lustre: lustre-MDT0000: new disk, initializing [ 6821.461467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6821.463970] Lustre: Skipped 1 previous similar message [ 6821.470511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6822.741758] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6826.832731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6826.857123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6826.891277] Lustre: 333674:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6826.914505] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6826.917117] Lustre: Skipped 1 previous similar message [ 6826.965872] Lustre: lustre-MDT0001: new disk, initializing [ 6827.016380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6827.019609] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6828.304563] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6831.325383] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6832.472360] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6834.579380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6834.601993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6834.692515] Lustre: lustre-OST0000: new disk, initializing [ 6834.694770] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6836.409177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6836.413135] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6836.450167] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6836.491223] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6839.483779] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6840.787358] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6840.851089] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6842.074365] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6842.141247] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6846.945784] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6846.946489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6846.948319] LustreError: Skipped 1 previous similar message [ 6846.948784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6846.954684] Lustre: Skipped 4 previous similar messages [ 6846.959117] Lustre: Skipped 5 previous similar messages [ 6852.066477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6852.069465] Lustre: Skipped 1 previous similar message [ 6857.183239] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6857.209522] LustreError: 336613:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6857.211416] LustreError: 336613:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6857.278786] Lustre: server umount lustre-OST0000 complete [ 6862.306306] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6862.307639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6862.310594] Lustre: Skipped 1 previous similar message [ 6862.315882] Lustre: Skipped 3 previous similar messages [ 6864.663245] Lustre: server umount lustre-MDT0000 complete [ 6865.901531] LustreError: 334456:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760132214 with bad export cookie 18002641596697769677 [ 6865.905410] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6865.907542] LustreError: 334456:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6866.056799] Lustre: server umount lustre-MDT0001 complete [ 6868.677731] XFS (loop0): Unmounting Filesystem [ 6870.272988] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 17:36:59 (1760132219) [ 6872.576591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6872.723483] LustreError: 337832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6872.747463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6872.749534] Lustre: Skipped 2 previous similar messages [ 6874.112936] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6877.041618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6878.480442] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6880.292907] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6881.539512] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6883.799186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6885.932190] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6887.727730] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6907.872939] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6907.876975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6907.881935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6912.747197] LustreError: 340196:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6912.749717] LustreError: 340196:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6912.812569] Lustre: server umount lustre-OST0000 complete [ 6928.351123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6928.417609] Lustre: server umount lustre-MDT0000 complete [ 6929.683355] LustreError: 337813:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760132278 with bad export cookie 18002641596697770559 [ 6929.688417] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6929.689123] LustreError: 337813:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6929.837124] Lustre: server umount lustre-MDT0001 complete [ 6933.623497] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 6934.755945] Key type lgssc unregistered [ 6934.898493] LNet: 341131:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6934.900807] LNetError: 341131:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6934.907300] LNet: Removed LNI 192.168.204.112@tcp [ 6935.238136] Key type .llcrypt unregistered [ 6935.239182] Key type ._llcrypt unregistered [ 6945.529096] Key type ._llcrypt registered [ 6945.530094] Key type .llcrypt registered [ 6945.567172] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 6945.904331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6945.961775] alg: No test for adler32 (adler32-zlib) [ 6946.830431] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 6946.924587] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 6948.511195] Key type lgssc registered [ 6948.893459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6951.499037] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6951.506383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6952.668851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6953.952829] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6956.777554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6956.896947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6958.184032] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6960.030525] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6961.273890] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6963.616259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6963.734211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6965.635412] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6967.556594] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6982.623765] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6982.625885] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6982.629380] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6983.138224] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6983.138242] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6986.874151] LustreError: 344974:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6986.897437] Lustre: server umount lustre-MDT0001 complete [ 6987.850851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6988.256386] LustreError: 342655:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6988.261678] LustreError: 342655:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6990.416429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6990.446169] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6990.447809] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6990.458756] Lustre: 342644:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6990.576144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6991.879270] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6995.943648] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6997.476739] LustreError: 346051:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 6997.479372] LustreError: 346051:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 6997.481886] LustreError: 342646:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 6997.499029] LustreError: 346055:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.112@tcp: cfg command failed: rc = -17 [ 6997.504431] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 6997.504431] [ 7295.719817] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.12@tcp (stopping) [ 7295.721811] Lustre: Skipped 2 previous similar messages [ 7298.017568] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7298.018446] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7298.021994] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7298.024983] Lustre: Skipped 1 previous similar message [ 7303.136332] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7303.138105] Lustre: Skipped 2 previous similar messages [ 7308.255123] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7308.292659] LustreError: 357593:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7308.294413] LustreError: 357593:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7308.324841] Lustre: server umount lustre-MDT0001 complete [ 7309.269106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7311.071524] LustreError: 344817:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7311.075936] LustreError: 344817:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7311.788533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7311.817548] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7311.819437] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7311.831246] Lustre: 342643:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7311.957958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7313.296505] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 7316.965901] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7320.038301] LustreError: 358771:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 7320.040683] LustreError: 358771:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 7320.043149] LustreError: 342646:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 7320.046254] LustreError: 342646:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 7320.066599] LustreError: 358774:0:(obd_config.c:689:class_setup()) Device 8 already setup (type osp) [ 7320.068473] LustreError: 358774:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.112@tcp: cfg command failed: rc = -17 [ 7320.070701] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.204.112@tcp [ 7320.070701] [ 7616.993895] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7616.997292] Lustre: Skipped 1 previous similar message [ 7616.999150] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7617.000857] Lustre: Skipped 4 previous similar messages [ 7622.530478] LustreError: 370309:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7622.533887] LustreError: 370309:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7622.560483] Lustre: server umount lustre-MDT0001 complete [ 7623.391556] LustreError: 342650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7623.556682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7625.980944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7626.011938] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7626.013426] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7626.025337] Lustre: 342644:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7626.151421] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 7627.366474] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 7631.332575] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7632.868919] LustreError: 371436:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 7632.871346] LustreError: 371436:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 7632.873824] LustreError: 342646:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 7632.875725] LustreError: 342646:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 7632.894043] LustreError: 371439:0:(lod_lov.c:207:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 7632.898750] LustreError: 371439:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.112@tcp: cfg command failed: rc = -114 [ 7632.901512] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 7632.901512] [ 7943.649502] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7943.650986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7943.655441] Lustre: Skipped 2 previous similar messages [ 7943.656920] Lustre: Skipped 3 previous similar messages [ 7943.661664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7947.775906] LustreError: 383094:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7947.779095] LustreError: 383094:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7947.815963] Lustre: server umount lustre-MDT0000 complete [ 7948.770747] LustreError: 342651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7948.776899] LustreError: 342651:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7949.081375] LustreError: 344278:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760133297 with bad export cookie 10538327113216129990 [ 7949.084066] LustreError: 344278:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7949.087221] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7953.888175] LustreError: 342651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7953.888515] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7953.895565] Lustre: Skipped 1 previous similar message [ 7955.200478] LustreError: 383295:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7955.202314] LustreError: 383295:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7955.295927] Lustre: server umount lustre-MDT0001 complete [ 7956.663465] Lustre: server umount lustre-OST0000 complete [ 7959.552599] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 7962.741509] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 7966.638461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7968.800980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7970.694202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7972.664740] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7974.922254] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 17:55:23 (1760133323) [ 7977.034412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7977.060428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7977.154405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7977.168050] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7977.209558] Lustre: lustre-MDT0000: new disk, initializing [ 7977.237197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7977.243431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7978.528322] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 7982.457583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7982.478350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7982.506843] Lustre: 386024:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7982.537994] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7982.539138] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7982.539853] Lustre: Skipped 4 previous similar messages [ 7982.541355] Lustre: Skipped 1 previous similar message [ 7982.577944] Lustre: lustre-MDT0001: new disk, initializing [ 7982.611156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7982.623606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7982.626801] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7983.808189] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 7986.807566] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7987.922846] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7990.165830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7990.194705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7990.288566] Lustre: lustre-OST0000: new disk, initializing [ 7990.290650] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7990.324292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7991.673969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7991.677297] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7991.725040] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7992.206702] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 7995.240871] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7997.920033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7997.922677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8003.963872] LustreError: 388467:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8003.965678] LustreError: 388467:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 8003.998453] Lustre: server umount lustre-MDT0000 complete [ 8005.207432] LustreError: 386804:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760133353 with bad export cookie 10538327113216131950 [ 8005.210337] LustreError: 386804:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8005.212529] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8005.212721] LustreError: 386824:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [ 8005.359648] Lustre: server umount lustre-MDT0001 complete [ 8016.886608] Lustre: server umount lustre-OST0000 complete [ 8025.512635] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 8028.537792] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 8031.906145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8034.427991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8036.303718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8038.273527] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8038.832843] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7810 sec ========== 17:56:27 (1760133387) [ 8039.356541] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 17:56:28 (1760133388) === [ 8040.456378] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 17:56:29 (1760133389) === [ 8053.192863] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 8054.270728] Key type lgssc unregistered [ 8054.413512] LNet: 393970:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8054.415759] LNetError: 393970:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8054.423385] LNet: Removed LNI 192.168.204.112@tcp [ 8054.738322] Key type .llcrypt unregistered [ 8054.739250] Key type ._llcrypt unregistered