[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 463891232 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002375] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007478] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010141] LSM: Security Framework initializing [ 0.011056] Yama: becoming mindful. [ 0.012043] SELinux: Initializing. [ 0.013069] *** VALIDATE selinux *** [ 0.022036] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027447] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028160] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029123] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030117] *** VALIDATE tmpfs *** [ 0.031472] *** VALIDATE proc *** [ 0.033247] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.036151] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037134] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039028] Spectre V2 : User space: Vulnerable [ 0.040008] Speculative Store Bypass: Vulnerable [ 0.043558] debug: unmapping init [mem 0xffffffff94859000-0xffffffff94860fff] [ 0.045193] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046705] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047023] ... version: 2 [ 0.048011] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050013] ... value mask: 0000ffffffffffff [ 0.051016] ... max period: 00007fffffffffff [ 0.052014] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.054272] rcu: Hierarchical SRCU implementation. [ 0.056502] smp: Bringing up secondary CPUs ... [ 0.057487] x86: Booting SMP configuration: [ 0.058016] .... node #0, CPUs: #1 #2 #3 [ 0.061088] smp: Brought up 1 node, 4 CPUs [ 0.063013] smpboot: Max logical packages: 1 [ 0.064018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.150045] node 0 deferred pages initialised in 84ms [ 0.153123] devtmpfs: initialized [ 0.154311] x86/mm: Memory block size: 128MB [ 0.156801] gcov: version magic: 0x41383552 [ 0.158303] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159076] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.160240] pinctrl core: initialized pinctrl subsystem [ 0.161224] [ 0.161710] ************************************************************* [ 0.162014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163014] ** ** [ 0.164008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165012] ** ** [ 0.166012] ** This means that this kernel is built to expose internal ** [ 0.167010] ** IOMMU data structures, which may compromise security on ** [ 0.168009] ** your system. ** [ 0.169015] ** ** [ 0.170013] ** If you see this message and you are not debugging the ** [ 0.171016] ** kernel, report this immediately to your vendor! ** [ 0.172015] ** ** [ 0.173021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.174016] ************************************************************* [ 0.175788] NET: Registered protocol family 16 [ 0.176457] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.177087] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.178072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.179507] cpuidle: using governor menu [ 0.181792] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.182649] PCI: Using configuration type 1 for base access [ 0.186266] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.201047] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.202022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.203173] cryptd: max_cpu_qlen set to 1000 [ 0.206789] ACPI: Added _OSI(Module Device) [ 0.208016] ACPI: Added _OSI(Processor Device) [ 0.210014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.212016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.216000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.224670] ACPI: Interpreter enabled [ 0.226083] ACPI: PM: (supports S0 S3 S4 S5) [ 0.228013] ACPI: Using IOAPIC for interrupt routing [ 0.230140] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.233727] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.245392] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.247051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.250023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.253088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.258682] acpiphp: Slot [2] registered [ 0.261131] acpiphp: Slot [5] registered [ 0.262129] acpiphp: Slot [6] registered [ 0.264112] acpiphp: Slot [7] registered [ 0.265122] acpiphp: Slot [8] registered [ 0.266121] acpiphp: Slot [9] registered [ 0.268138] acpiphp: Slot [10] registered [ 0.269108] acpiphp: Slot [3] registered [ 0.271111] acpiphp: Slot [4] registered [ 0.272110] acpiphp: Slot [11] registered [ 0.274096] acpiphp: Slot [12] registered [ 0.275090] acpiphp: Slot [13] registered [ 0.276194] acpiphp: Slot [14] registered [ 0.278109] acpiphp: Slot [15] registered [ 0.280103] acpiphp: Slot [16] registered [ 0.281106] acpiphp: Slot [17] registered [ 0.283110] acpiphp: Slot [18] registered [ 0.284100] acpiphp: Slot [19] registered [ 0.286109] acpiphp: Slot [20] registered [ 0.287149] acpiphp: Slot [21] registered [ 0.289109] acpiphp: Slot [22] registered [ 0.290077] acpiphp: Slot [23] registered [ 0.291088] acpiphp: Slot [24] registered [ 0.294103] acpiphp: Slot [25] registered [ 0.295100] acpiphp: Slot [26] registered [ 0.296136] acpiphp: Slot [27] registered [ 0.297103] acpiphp: Slot [28] registered [ 0.298084] acpiphp: Slot [29] registered [ 0.299082] acpiphp: Slot [30] registered [ 0.300082] acpiphp: Slot [31] registered [ 0.301077] PCI host bridge to bus 0000:00 [ 0.302016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.304021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.306071] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.307021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.309019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.311022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.312175] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.314911] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.317145] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.326013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.330052] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.333023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.335017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.336011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.338530] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.340722] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.342032] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.344694] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.348012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.359018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.363013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.367798] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.394021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.412016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.437017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.446728] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.453018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.459018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.474031] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.483586] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.490016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.500019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.513017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.522416] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.552021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.560014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.584013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.591502] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.599022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.606021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.620015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.630233] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.635016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.640016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.662019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.672418] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.674355] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.676402] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.679403] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.682239] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.687027] iommu: Default domain type: Passthrough [ 0.688349] SCSI subsystem initialized [ 0.690134] ACPI: bus type USB registered [ 0.691081] usbcore: registered new interface driver usbfs [ 0.694073] usbcore: registered new interface driver hub [ 0.696078] usbcore: registered new device driver usb [ 0.697183] pps_core: LinuxPPS API ver. 1 registered [ 0.699010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.702062] PTP clock support registered [ 0.704081] EDAC MC: Ver: 3.0.0 [ 0.706192] PCI: Using ACPI for IRQ routing [ 0.707000] NetLabel: Initializing [ 0.708010] NetLabel: domain hash size = 128 [ 0.709006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.711077] NetLabel: unlabeled traffic allowed by default [ 0.713129] vgaarb: loaded [ 0.716245] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.718015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.728139] clocksource: Switched to clocksource kvm-clock [ 0.843493] VFS: Disk quotas dquot_6.6.0 [ 0.844875] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.847520] *** VALIDATE ramfs *** [ 0.848791] *** VALIDATE hugetlbfs *** [ 0.850500] pnp: PnP ACPI init [ 0.852836] pnp: PnP ACPI: found 6 devices [ 0.871294] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.874202] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.876200] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.878259] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.880532] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.882832] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.885679] NET: Registered protocol family 2 [ 0.888135] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.892713] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.897339] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.902567] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.906521] TCP: Hash tables configured (established 65536 bind 65536) [ 0.910375] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.913537] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.916474] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.920791] NET: Registered protocol family 1 [ 0.923898] RPC: Registered named UNIX socket transport module. [ 0.925683] RPC: Registered udp transport module. [ 0.927360] RPC: Registered tcp transport module. [ 0.928852] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.931284] NET: Registered protocol family 44 [ 0.933078] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.935544] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.937765] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.940427] PCI: CLS 0 bytes, default 64 [ 0.942163] Unpacking initramfs... [ 2.306819] debug: unmapping init [mem 0xffff9b6cfcc54000-0xffff9b6cfffbffff] [ 2.310566] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.312788] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.315572] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.783557] Initialise system trusted keyrings [ 2.784629] Key type blacklist registered [ 2.785836] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.796682] zbud: loaded [ 2.799690] *** VALIDATE nfs *** [ 2.800757] *** VALIDATE nfs4 *** [ 2.802318] pstore: using deflate compression [ 2.805060] Platform Keyring initialized [ 2.889367] NET: Registered protocol family 38 [ 2.890710] Key type asymmetric registered [ 2.891826] Asymmetric key parser 'x509' registered [ 2.893448] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.896128] io scheduler mq-deadline registered [ 2.897722] io scheduler kyber registered [ 2.898880] io scheduler bfq registered [ 2.901240] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.903317] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.905252] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.907233] ACPI: Power Button [PWRF] [ 2.911175] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.915815] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.934438] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.943403] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.969695] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.995655] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.025802] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.030800] Non-volatile memory driver v1.3 [ 3.032485] Linux agpgart interface v0.103 [ 3.061766] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.064602] vda: detected capacity change from 0 to 74702848 [ 3.091803] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.095082] vdb: detected capacity change from 0 to 1073741824 [ 3.115320] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.118267] vdc: detected capacity change from 0 to 2621440000 [ 3.138639] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.141571] vdd: detected capacity change from 0 to 2621440000 [ 3.157603] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.159289] vde: detected capacity change from 0 to 4294967296 [ 3.175280] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.178771] vdf: detected capacity change from 0 to 4294967296 [ 3.185451] libphy: Fixed MDIO Bus: probed [ 3.193388] usbcore: registered new interface driver usbserial_generic [ 3.195422] usbserial: USB Serial support registered for generic [ 3.197869] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.202521] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.204538] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.206830] mousedev: PS/2 mouse device common for all mice [ 3.208665] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.212285] rtc_cmos 00:05: RTC can wake from S4 [ 3.215562] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.218679] rtc_cmos 00:05: registered as rtc0 [ 3.220358] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.220459] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.229959] intel_pstate: CPU model not supported [ 3.238642] hid: raw HID events driver (C) Jiri Kosina [ 3.240692] usbcore: registered new interface driver usbhid [ 3.242627] usbhid: USB HID core driver [ 3.244365] drop_monitor: Initializing network drop monitor service [ 3.246148] Initializing XFRM netlink socket [ 3.247902] NET: Registered protocol family 10 [ 3.250581] Segment Routing with IPv6 [ 3.251419] NET: Registered protocol family 17 [ 3.253131] mpls_gso: MPLS GSO support [ 3.259042] RAS: Correctable Errors collector initialized. [ 3.260227] AVX version of gcm_enc/dec engaged. [ 3.261097] AES CTR mode by8 optimization enabled [ 3.314691] sched_clock: Marking stable (3314620682, 0)->(4144640971, -830020289) [ 3.316762] registered taskstats version 1 [ 3.318335] Loading compiled-in X.509 certificates [ 3.319691] zswap: loaded using pool lzo/zbud [ 3.337518] Key type big_key registered [ 3.346813] Key type encrypted registered [ 3.348423] ima: No TPM chip found, activating TPM-bypass! [ 3.350315] ima: Allocated hash algorithm: sha1 [ 3.352040] ima: No architecture policies found [ 3.353276] evm: Initialising EVM extended attributes: [ 3.354413] evm: security.selinux [ 3.355215] evm: security.ima [ 3.355882] evm: security.capability [ 3.356669] evm: HMAC attrs: 0x1 [ 3.358957] rtc_cmos 00:05: setting system clock to 2026-08-15 16:50:15 UTC (1786812615) [ 3.363181] debug: unmapping init [mem 0xffffffff95803000-0xffffffff959fffff] [ 3.365172] debug: unmapping init [mem 0xffffffff94582000-0xffffffff94858fff] [ 3.373110] Write protecting the kernel read-only data: 28672k [ 3.375389] debug: unmapping init [mem 0xffffffff92c03000-0xffffffff92dfffff] [ 3.377496] debug: unmapping init [mem 0xffffffff93514000-0xffffffff935fffff] [ 3.410358] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.416864] systemd[1]: Detected virtualization kvm. [ 3.418454] systemd[1]: Detected architecture x86-64. [ 3.419734] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.444768] systemd[1]: No hostname configured. [ 3.446794] systemd[1]: Set hostname to . [ 3.449021] random: systemd: uninitialized urandom read (16 bytes read) [ 3.451689] systemd[1]: Initializing machine ID from random generator. [ 3.497946] random: ln: uninitialized urandom read (6 bytes read) [ 3.562660] random: systemd: uninitialized urandom read (16 bytes read) [ 3.565586] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.570484] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.575501] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... [ 3.688166] random: fast init done [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.081324] device-mapper: uevent: version 1.0.3 [ 4.083811] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.703415] virtio_net virtio0 ens2: renamed from eth0 [ 4.759276] scsi host0: ata_piix [ 4.761829] scsi host1: ata_piix [ 4.764217] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.767509] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.080737] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.865338] random: crng init done [ 9.866782] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.132275] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.176849] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.464526] SELinux: Disabled at runtime. [ 11.533776] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.542066] systemd[1]: Detected virtualization kvm. [ 11.544083] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.077729] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.080953] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.084794] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.088123] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.093302] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.104223] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.114936] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK [ 12.174747] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.579321] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.930224] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.955056] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.175739] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.184860] EDAC sbridge: Ver: 1.1.2 [ 14.731973] Key type dns_resolver registered [ 15.034348] NFS: Registering the id_resolver key type [ 15.036252] Key type id_resolver registered [ 15.037902] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg104-server login: [ 41.681692] libcfs: loading out-of-tree module taints kernel. [ 41.702537] Key type ._llcrypt registered [ 41.704279] Key type .llcrypt registered [ 41.757459] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 50.669975] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 52.109579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.136093] alg: No test for adler32 (adler32-zlib) [ 53.643899] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 54.606624] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 56.391320] Key type lgssc registered [ 58.774796] Lustre: Echo OBD driver; http://www.lustre.org/ [ 75.328039] hrtimer: interrupt took 4943772 ns [ 77.454501] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 125.293905] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 141.095040] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 141.137804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 142.577766] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 142.603382] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 142.768601] Lustre: lustre-MDT0000: new disk, initializing [ 142.947720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 142.980633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 148.752947] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 165.105333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 165.231562] Lustre: 6484:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 165.294959] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 165.304488] Lustre: Skipped 1 previous similar message [ 165.434815] Lustre: lustre-MDT0001: new disk, initializing [ 165.502580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 165.536088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 165.548476] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 170.449665] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 175.714449] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 187.123621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 187.501669] Lustre: lustre-OST0000: new disk, initializing [ 187.506300] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 187.510667] Lustre: 8422:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 187.581904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 189.741244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 189.756711] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 189.877530] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 193.739563] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.339913] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.466887] Lustre: lustre-OST0001: new disk, initializing [ 209.478546] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 209.484982] Lustre: 9499:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 209.586369] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 216.966775] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.705186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 219.714213] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 219.760269] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 229.466368] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 236.917644] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 243.898977] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing check_logdir /tmp/testlogs/ [ 248.261277] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing yml_node [ 252.689346] Lustre: DEBUG MARKER: Client: 2.17.57.2 [ 255.100351] Lustre: DEBUG MARKER: MDS: 2.17.57.2 [ 257.920654] Lustre: DEBUG MARKER: OSS: 2.17.57.2 [ 259.425314] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 15 12:54:30 EDT 2026 [ 275.168332] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 276.802501] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 293.857939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 293.865474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 293.886182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 296.418298] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 296.419532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 296.443602] Lustre: Skipped 1 previous similar message [ 296.453273] Lustre: Skipped 3 previous similar messages [ 298.987068] Lustre: server umount lustre-MDT0000 complete [ 302.336844] LustreError: 6476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786812914 with bad export cookie 1606536899300504881 [ 302.342924] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 302.347986] LustreError: 6476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 306.664363] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.669519] LustreError: 6492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 306.675111] Lustre: Skipped 1 previous similar message [ 306.687486] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 306.688669] LustreError: 6492:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 306.715698] Lustre: Skipped 1 previous similar message [ 307.693604] LustreError: 6492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 307.721269] LustreError: 6492:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 308.807833] Lustre: server umount lustre-MDT0001 complete [ 314.735333] Lustre: server umount lustre-OST0000 complete [ 319.010755] Lustre: server umount lustre-OST0001 complete [ 327.921140] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 339.467492] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 375.015934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 375.233394] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 375.261883] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 375.335263] Lustre: lustre-MDT0000: new disk, initializing [ 375.393958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 375.406563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 378.617894] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 387.221973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 387.349543] Lustre: 15050:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 387.405552] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 387.408853] Lustre: Skipped 1 previous similar message [ 387.507143] Lustre: lustre-MDT0001: new disk, initializing [ 387.616648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 387.645677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 387.654546] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 392.043452] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 401.365418] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 407.367166] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 415.125032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 415.518956] Lustre: lustre-OST0000: new disk, initializing [ 415.523675] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 415.540191] Lustre: 17020:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 415.671850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 417.302810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 417.331482] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 417.487510] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 422.370948] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 431.427248] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 436.984647] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 437.296111] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 442.686940] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 443.027837] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 448.484431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 448.503908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 448.531722] Lustre: Skipped 1 previous similar message [ 448.554476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 448.557080] Lustre: Skipped 2 previous similar messages [ 451.477209] Lustre: server umount lustre-OST0000 complete [ 458.721448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 458.767535] Lustre: Skipped 1 previous similar message [ 458.773605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 458.785429] Lustre: Skipped 1 previous similar message [ 462.447718] Lustre: server umount lustre-MDT0000 complete [ 463.841522] LustreError: 15062:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 463.852116] LustreError: 15062:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 465.845097] LustreError: 15043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813077 with bad export cookie 1606536899300506407 [ 465.850694] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 465.858271] LustreError: 15043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 466.293748] Lustre: server umount lustre-MDT0001 complete [ 470.562915] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 12:58:02 (1786813082) [ 476.509864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 476.985541] LustreError: 19283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 477.074124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 481.434835] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 482.274433] LustreError: 19283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 482.300708] LustreError: 19283:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 489.688832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 494.956603] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 502.956381] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 508.886531] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 515.896664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 522.752360] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 531.607989] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 538.153580] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 546.280237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 546.282216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 546.306746] Lustre: Skipped 3 previous similar messages [ 546.325695] Lustre: Skipped 2 previous similar messages [ 551.888896] Lustre: server umount lustre-MDT0000 complete [ 555.956252] LustreError: 21813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813168 with bad export cookie 1606536899300507289 [ 555.964304] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 555.968684] LustreError: 21813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 556.229895] Lustre: server umount lustre-MDT0001 complete [ 570.102870] Lustre: server umount lustre-OST0000 complete [ 590.978669] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 601.238762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 601.494856] LustreError: 23665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 601.518990] LustreError: 23665:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 601.572732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 601.586056] Lustre: Skipped 2 previous similar messages [ 606.345916] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 615.895959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 621.954876] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 624.937120] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 632.299643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 639.323385] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 649.614340] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 649.952427] Lustre: lustre-OST0001: new disk, initializing [ 649.956624] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 649.961600] Lustre: 25993:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 651.308581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 651.320325] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 651.394662] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 656.318218] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 665.635847] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 668.884565] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 681.955187] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 681.956393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 681.961641] Lustre: Skipped 3 previous similar messages [ 681.964930] Lustre: Skipped 5 previous similar messages [ 688.072109] Lustre: server umount lustre-MDT0000 complete [ 692.204320] LustreError: 23666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 692.234184] LustreError: 23666:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 692.524368] LustreError: 24404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813304 with bad export cookie 1606536899300508143 [ 692.532696] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 692.540948] LustreError: 24404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 692.999766] Lustre: server umount lustre-MDT0001 complete [ 707.618737] Lustre: server umount lustre-OST0000 complete [ 722.307055] Lustre: server umount lustre-OST0001 complete [ 740.624450] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 750.654398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 751.128948] LustreError: 29015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 751.213287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 751.219490] Lustre: Skipped 3 previous similar messages [ 756.086298] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 766.827730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 773.999503] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 777.073322] Lustre: 30165:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 783.936927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 791.414928] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 801.386460] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 807.686943] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 816.144357] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 820.215186] Lustre: 32044:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 827.374927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 827.380325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 827.401265] Lustre: Skipped 3 previous similar messages [ 827.404143] Lustre: Skipped 7 previous similar messages [ 833.621849] Lustre: server umount lustre-MDT0000 complete [ 837.096119] LustreError: 28997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813449 with bad export cookie 1606536899300509438 [ 837.101506] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 837.103995] LustreError: 28997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 837.567732] Lustre: server umount lustre-MDT0001 complete [ 851.094217] Lustre: server umount lustre-OST0000 complete [ 852.960139] Lustre: 3618:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786813449/real 1786813449] req@ffff9b6d5fd3d500 x1873608878358016/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786813465 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 854.684451] Lustre: server umount lustre-OST0001 complete [ 864.691523] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 13:04:36 (1786813476) [ 896.194986] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 907.918720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 908.354233] LustreError: 34846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 908.378973] LustreError: 34846:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 913.468174] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 921.533787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 926.683411] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 929.528938] Lustre: 35992:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 938.957285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 948.930525] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 958.290818] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 965.339170] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 973.466326] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 977.616199] Lustre: 37873:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 984.549028] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 984.557080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 984.560750] Lustre: Skipped 3 previous similar messages [ 984.574373] Lustre: Skipped 5 previous similar messages [ 990.600269] Lustre: server umount lustre-MDT0000 complete [ 994.330510] LustreError: 34829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813606 with bad export cookie 1606536899300510544 [ 994.332352] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 994.339130] LustreError: 34829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 994.699790] Lustre: server umount lustre-MDT0001 complete [ 1009.345472] Lustre: server umount lustre-OST0000 complete [ 1010.145782] Lustre: 3617:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786813606/real 1786813606] req@ffff9b6d7cb97800 x1873608878408704/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786813622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1013.299750] Lustre: server umount lustre-OST0001 complete [ 1022.280966] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 13:07:13 (1786813633) [ 1039.949561] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 1050.989458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1051.352264] LustreError: 40013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1051.384759] LustreError: 40013:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 1051.482440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1051.497252] Lustre: Skipped 7 previous similar messages [ 1055.483107] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1065.295592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1072.456623] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1076.755574] Lustre: 41162:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1083.570463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1090.282718] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1098.685504] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1105.060085] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1111.568218] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1115.096578] Lustre: 43039:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1115.847800] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1115.853742] Lustre: Skipped 1 previous similar message [ 1119.715825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1119.727601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1119.728120] Lustre: Skipped 3 previous similar messages [ 1119.737808] Lustre: Skipped 10 previous similar messages [ 1124.335483] Lustre: server umount lustre-MDT0000 complete [ 1127.856053] LustreError: 39995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813739 with bad export cookie 1606536899300511643 [ 1127.862499] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1127.866222] LustreError: 39995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1142.239388] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1142.717410] Lustre: server umount lustre-MDT0001 complete [ 1152.862533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1158.667894] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1167.666805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1169.202747] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1171.746671] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1179.875922] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1181.505435] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1187.306218] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1189.112072] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1200.535548] Lustre: server umount lustre-MDT0000 complete [ 1204.303143] LustreError: 39995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813816 with bad export cookie 1606536899300512749 [ 1204.307652] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1204.313459] LustreError: 39995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1204.755270] Lustre: server umount lustre-MDT0001 complete [ 1218.614889] Lustre: server umount lustre-OST0000 complete [ 1231.918104] Lustre: server umount lustre-OST0001 complete [ 1239.515670] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 13:10:51 (1786813851) [ 1255.115827] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 1264.910105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1269.741965] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1277.192113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1281.850562] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1284.730090] Lustre: 48804:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1291.029037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1296.602756] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1304.380047] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1310.854759] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1318.047196] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1321.932231] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1333.728825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1333.739448] LustreError: Skipped 1 previous similar message [ 1335.253927] Lustre: server umount lustre-MDT0000 complete [ 1335.787751] LustreError: 47645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1335.804797] LustreError: 47645:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 33 previous similar messages [ 1339.013365] LustreError: 47630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813951 with bad export cookie 1606536899300513568 [ 1339.021120] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1339.023913] LustreError: 47630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1339.385495] Lustre: server umount lustre-MDT0001 complete [ 1353.408787] Lustre: server umount lustre-OST0000 complete [ 1366.222979] Lustre: server umount lustre-OST0001 complete [ 1374.693342] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 13:13:06 (1786813986) [ 1376.888375] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1378.582691] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 13:13:10 (1786813990) [ 1389.034426] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 1397.476656] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 1419.257549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1419.438808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1419.495579] Lustre: lustre-MDT0000: new disk, initializing [ 1419.558895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1423.015874] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1431.458638] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1436.926708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1437.143969] Lustre: lustre-OST0000: new disk, initializing [ 1437.147049] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1437.150574] Lustre: Skipped 1 previous similar message [ 1437.156294] Lustre: 54878:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1438.351826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1438.360793] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1438.393517] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1442.701258] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1450.794330] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1452.770526] Lustre: DEBUG MARKER: create test files [ 1479.137344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1479.150198] Lustre: Skipped 12 previous similar messages [ 1479.155066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1479.161273] Lustre: Skipped 19 previous similar messages [ 1480.028703] Lustre: server umount lustre-OST0000 complete [ 1482.863560] Lustre: server umount lustre-MDT0000 complete [ 1490.910831] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 1493.092833] Key type lgssc unregistered [ 1493.326448] LNet: 56508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1493.338028] LNetError: 56508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1493.359591] LNet: Removed LNI 192.168.201.104@tcp [ 1494.196653] Key type .llcrypt unregistered [ 1494.199511] Key type ._llcrypt unregistered [ 1516.745614] Key type ._llcrypt registered [ 1516.749494] Key type .llcrypt registered [ 1516.910869] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 1518.339432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1518.366236] alg: No test for adler32 (adler32-zlib) [ 1519.355150] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 1519.584200] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 1521.279165] Key type lgssc registered [ 1522.298113] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1529.158600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1529.187479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1530.669738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1534.452129] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1540.281656] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1545.124113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1545.443904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1552.234254] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1555.718502] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1555.878076] Lustre: 59224:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1555.885511] Lustre: 59224:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1555.889980] Lustre: 59224:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1555.899141] Lustre: 59224:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1555.906544] Lustre: 59224:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1555.917734] Lustre: 59224:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1561.610387] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1567.121716] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1572.425979] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1581.538042] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1581.546301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1581.572411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1584.076961] Lustre: server umount lustre-OST0000 complete [ 1588.046502] Lustre: server umount lustre-MDT0000 complete [ 1599.307839] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 1601.916520] Key type lgssc unregistered [ 1602.286543] LNet: 60839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1602.297698] LNetError: 60839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1602.334447] LNet: Removed LNI 192.168.201.104@tcp [ 1603.617179] Key type .llcrypt unregistered [ 1603.621216] Key type ._llcrypt unregistered [ 1636.295801] Key type ._llcrypt registered [ 1636.297772] Key type .llcrypt registered [ 1636.404688] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 1637.868932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1637.906074] alg: No test for adler32 (adler32-zlib) [ 1639.016949] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 1639.374645] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 1641.047187] Key type lgssc registered [ 1642.254355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1649.098107] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1649.140843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1650.577054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1654.148366] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1662.817951] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1671.516959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1671.912580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1675.002827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1675.093933] Lustre: 63677:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1675.098977] Lustre: 63677:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1675.108417] Lustre: 63677:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1675.113087] Lustre: 63677:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1675.123857] Lustre: 63677:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1675.129549] Lustre: 63677:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1679.079621] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1685.859984] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1691.143508] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1731.556489] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1731.561642] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1731.571402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1734.627293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1736.267318] Lustre: server umount lustre-OST0000 complete [ 1740.749636] Lustre: server umount lustre-MDT0000 complete [ 1751.699171] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 1754.921348] Key type lgssc unregistered [ 1755.382836] LNet: 65274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1755.389562] LNetError: 65274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1755.406055] LNet: Removed LNI 192.168.201.104@tcp [ 1756.420507] Key type .llcrypt unregistered [ 1756.424659] Key type ._llcrypt unregistered [ 1776.591346] Key type ._llcrypt registered [ 1776.593981] Key type .llcrypt registered [ 1776.751898] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 1797.135704] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 1798.955311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1799.044188] alg: No test for adler32 (adler32-zlib) [ 1800.191283] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 1800.511554] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 1802.223224] Key type lgssc registered [ 1803.502587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1844.188877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1844.229672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1845.700611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1845.780816] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1845.900659] Lustre: lustre-MDT0000: new disk, initializing [ 1845.968434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1845.987507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1849.915765] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1862.267792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1862.450471] Lustre: 68432:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1862.566971] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1862.571166] Lustre: Skipped 1 previous similar message [ 1862.635475] Lustre: lustre-MDT0001: new disk, initializing [ 1862.734159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1862.772993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1862.798387] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1867.376981] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1877.326718] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1885.032239] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1892.994565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1893.148559] Lustre: lustre-OST0000: new disk, initializing [ 1893.151546] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1893.161262] Lustre: 70402:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1893.281545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1900.278544] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 1901.657560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1901.662911] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1901.831039] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1913.805749] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1921.657916] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1921.959481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1928.026703] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1928.325859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1932.782715] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1932.787580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1932.791751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1932.802222] Lustre: Skipped 1 previous similar message [ 1932.814231] Lustre: Skipped 1 previous similar message [ 1936.586657] Lustre: server umount lustre-OST0000 complete [ 1943.009110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1943.030537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1943.039029] Lustre: Skipped 1 previous similar message [ 1947.037533] Lustre: server umount lustre-MDT0000 complete [ 1948.131954] LustreError: 70148:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1948.149131] LustreError: 70148:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1951.115375] LustreError: 68425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814563 with bad export cookie 945498313093916563 [ 1951.124438] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1951.125499] LustreError: 68425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1951.527497] Lustre: server umount lustre-MDT0001 complete [ 1959.630161] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 13:22:50 (1786814570) [ 1973.778793] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 2021.681497] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 2032.510766] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 2075.268291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2075.547757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2075.581323] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2075.689495] Lustre: lustre-MDT0000: new disk, initializing [ 2075.812703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2075.829726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2080.428278] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2092.627884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2092.757601] Lustre: 76436:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2092.809860] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2092.816702] Lustre: Skipped 1 previous similar message [ 2092.897856] Lustre: lustre-MDT0001: new disk, initializing [ 2093.034280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2093.089414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2093.101310] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2097.984784] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2107.475856] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2114.237751] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2121.067181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2121.314614] Lustre: lustre-OST0000: new disk, initializing [ 2121.317235] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2121.320878] Lustre: 78404:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2121.392328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2126.912949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2126.927620] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2127.019745] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2127.038818] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2137.396685] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2142.894791] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2143.134988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2148.064811] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2148.253392] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2152.928128] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2152.932720] LustreError: Skipped 1 previous similar message [ 2152.935037] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2152.943673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2152.964914] Lustre: Skipped 2 previous similar messages [ 2156.628647] Lustre: server umount lustre-OST0000 complete [ 2163.169597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2163.178638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2163.182727] Lustre: Skipped 1 previous similar message [ 2163.205952] Lustre: Skipped 2 previous similar messages [ 2167.096927] Lustre: server umount lustre-MDT0000 complete [ 2168.298285] LustreError: 76442:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2168.324125] LustreError: 76442:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2170.759731] LustreError: 76427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814782 with bad export cookie 945498313093917445 [ 2170.765800] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2170.774483] LustreError: 76427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2170.991505] Lustre: server umount lustre-MDT0001 complete [ 2180.126912] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 13:26:31 (1786814791) [ 2187.538369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2188.295063] LustreError: 80815:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2188.352940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2192.993085] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2193.376940] LustreError: 80816:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2198.527592] LustreError: 80815:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2202.200137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2207.178274] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2214.385676] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2220.613915] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2227.801056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2228.109705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2228.119711] Lustre: Skipped 1 previous similar message [ 2235.015984] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2243.397893] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2247.281048] Lustre: 83485:0:(genops.c:1795:obd_export_evict_by_uuid()) MGS: evicting d99e98d3-5dab-41ee-bbd7-95d49efaf48a at adminstrative request [ 2247.599410] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2247.627583] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2256.865082] Lustre: MGS: Client d99e98d3-5dab-41ee-bbd7-95d49efaf48a (at 0@lo) reconnecting [ 2256.887158] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0xd1f15b4cfb1ae77 to 0xd1f15b4cfb1b117 [ 2256.914283] Lustre: MGC192.168.201.104@tcp: Connection restored to 0@lo (at 0@lo) [ 2286.740835] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2287.091322] Lustre: lustre-OST0001: new disk, initializing [ 2287.103489] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2287.113351] Lustre: 83880:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2292.928443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2292.941030] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2293.212839] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2295.352335] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2306.897870] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2313.711482] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2313.722350] LustreError: Skipped 1 previous similar message [ 2313.725066] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2313.741183] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2316.556310] Lustre: server umount lustre-OST0001 complete [ 2318.833508] LustreError: 83672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2318.858759] LustreError: 83672:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2321.395095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2321.402452] LustreError: Skipped 1 previous similar message [ 2323.938443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2323.941721] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2323.949589] Lustre: Skipped 2 previous similar messages [ 2323.965619] Lustre: Skipped 2 previous similar messages [ 2326.787849] Lustre: server umount lustre-OST0000 complete [ 2337.669778] Lustre: server umount lustre-MDT0000 complete [ 2339.299395] LustreError: 80810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2339.320062] LustreError: 80810:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2341.954498] LustreError: 83879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814954 with bad export cookie 945498313093918999 [ 2341.956317] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2341.974636] LustreError: 83879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2342.343624] Lustre: server umount lustre-MDT0001 complete [ 2351.892158] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 13:29:23 (1786814963) [ 2354.226468] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2356.249321] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 13:29:27 (1786814967) [ 2358.200455] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2361.042542] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 13:29:31 (1786814971) [ 2363.343270] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2365.731601] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 13:29:36 (1786814976) [ 2368.102290] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2370.218304] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 13:29:41 (1786814981) [ 2372.441685] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2378.538848] loop: module loaded [ 2378.558186] loop0: detected capacity change from 0 to 4294967296 [ 2378.608530] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2380.157938] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2384.500071] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2387.529811] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2415.728966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2415.779794] LustreError: 88005:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2415.790758] LustreError: 88005:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2415.799892] LustreError: 88005:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2415.807560] LustreError: 88005:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2434.067981] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 2445.269896] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 2481.630106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2481.963047] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2482.003153] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2482.125515] Lustre: lustre-MDT0000: new disk, initializing [ 2482.282556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2482.291524] Lustre: Skipped 1 previous similar message [ 2482.320742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2487.886118] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2501.067881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2501.296167] Lustre: 90870:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2501.837068] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2507.435780] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2518.248134] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2523.949379] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2530.430546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2530.645846] Lustre: 92841:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2537.355627] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2539.592767] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2539.733409] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2549.100274] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2554.442274] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2554.613413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2559.583604] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2559.939643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2565.602109] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2565.604845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2565.619126] LustreError: Skipped 1 previous similar message [ 2565.631902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2565.643937] Lustre: Skipped 3 previous similar messages [ 2565.678555] Lustre: Skipped 4 previous similar messages [ 2576.353849] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2576.730395] Lustre: server umount lustre-OST0000 complete [ 2595.296649] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2595.719202] Lustre: server umount lustre-MDT0000 complete [ 2596.322162] LustreError: 90878:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2596.350748] LustreError: 90878:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2600.559957] LustreError: 90864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815212 with bad export cookie 945498313093919615 [ 2600.562397] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2600.566230] LustreError: 90864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2600.963178] Lustre: server umount lustre-MDT0001 complete [ 2611.967112] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 13:33:42 (1786815222) [ 2620.871479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2621.889959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-60 [ 2621.892667] Lustre: Skipped 2 previous similar messages [ 2628.373363] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2637.473320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2644.028530] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2652.147423] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2657.983247] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2664.534927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2671.505258] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2679.477623] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2683.934778] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2684.192238] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2689.751642] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2690.188495] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2700.410571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2700.647352] Lustre: lustre-OST0001: new disk, initializing [ 2700.649961] Lustre: Skipped 2 previous similar messages [ 2700.654546] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2700.665473] Lustre: Skipped 3 previous similar messages [ 2700.685795] Lustre: 98625:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2706.188807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2706.195462] Lustre: Skipped 2 previous similar messages [ 2706.203472] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2706.361471] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2708.547437] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2717.997797] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2723.061200] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2723.335653] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2728.052090] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2728.211656] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2748.974779] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2772.170447] Lustre: Failing over lustre-MDT0000 [ 2772.607322] LustreError: 95251:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2772.620078] LustreError: 95251:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2772.898878] Lustre: server umount lustre-MDT0000 complete [ 2772.964685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2772.971200] Lustre: Skipped 2 previous similar messages [ 2794.464410] Lustre: 66656:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786815390/real 1786815390] req@0000000013cbcbe8 x1873610709165696/t0(0) o400->MGC192.168.201.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786815406 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2794.492905] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2798.023887] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2798.029231] LDISKFS-fs (dm-0): recovery complete [ 2798.041371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2804.733616] LustreError: 66653:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000097205f3f x1873610709174400/t0(0) o250->MGC192.168.201.104@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2805.287725] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2809.418079] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2809.429364] Lustre: 95252:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2810.358274] Lustre: 95984:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2810.375057] Lustre: 95984:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 1 previous similar message [ 2810.395647] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2810.543090] LustreError: 101480:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2810.872032] LustreError: 101480:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2811.210832] LustreError: 101480:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2811.217485] LustreError: 101480:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 1 previous similar message [ 2811.560098] LustreError: 101480:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2811.570617] LustreError: 101480:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 1 previous similar message [ 2811.608099] Lustre: 101480:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2811.635399] Lustre: 101480:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 7 previous similar messages [ 2812.068846] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2812.272678] LustreError: 101480:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2812.285388] LustreError: 101480:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 2 previous similar messages [ 2812.615808] LustreError: 101480:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2812.624101] LustreError: 101480:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 2 previous similar messages [ 2813.616468] Lustre: 101480:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2813.626678] Lustre: 101480:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 5 previous similar messages [ 2814.279488] LustreError: 101480:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2814.303335] LustreError: 101480:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 5 previous similar messages [ 2814.623477] LustreError: 101480:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2814.636500] LustreError: 101480:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 5 previous similar messages [ 2817.819672] Lustre: 101480:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2817.832901] Lustre: 101480:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 12 previous similar messages [ 2818.497498] LustreError: 101480:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2818.506748] LustreError: 101480:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 12 previous similar messages [ 2818.840822] LustreError: 101480:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2818.846286] LustreError: 101480:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 12 previous similar messages [ 2825.848543] Lustre: 101480:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2825.875418] Lustre: 101480:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 24 previous similar messages [ 2826.505360] LustreError: 101480:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2826.535092] LustreError: 101480:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 24 previous similar messages [ 2826.855342] LustreError: 101480:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2826.863960] LustreError: 101480:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 24 previous similar messages [ 2841.881200] Lustre: 101480:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2841.905113] Lustre: 101480:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 49 previous similar messages [ 2842.555259] LustreError: 101480:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2842.568366] LustreError: 101480:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 49 previous similar messages [ 2842.895808] LustreError: 101480:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2842.904547] LustreError: 101480:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 49 previous similar messages [ 2869.151216] Lustre: 101480:0:(ldlm_lib.c:2394:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2869.165559] Lustre: 101480:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2869.179824] Lustre: 101480:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client ca434dac-b499-4d12-a205-b77e88199c72@192.168.201.4@tcp [ 2869.198588] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2869.213614] Lustre: 101480:0:(ldlm_lib.c:1914:abort_req_replay_queue()) @@@ aborted: req@00000000ea47988b x1873610687977728/t0(8589934780) o35->ca434dac-b499-4d12-a205-b77e88199c72@192.168.201.4@tcp:26/0 lens 392/0 e 0 to 0 dl 1786815491 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2869.250716] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2869.269171] Lustre: lustre-MDT0000: Denying connection for new client ca434dac-b499-4d12-a205-b77e88199c72 (at 192.168.201.4@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2869.276722] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2869.332873] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2869.341450] Lustre: Skipped 2 previous similar messages [ 2869.351433] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2869.409587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:80 to 0x280000401:577) [ 2869.410933] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:48 to 0x2c0000401:545) [ 2885.603337] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2885.619738] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2885.631718] Lustre: Skipped 3 previous similar messages [ 2885.644909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2885.653905] Lustre: Skipped 10 previous similar messages [ 2891.545368] Lustre: server umount lustre-OST0000 complete [ 2902.502884] LustreError: 100030:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2902.538599] LustreError: 100030:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 52 previous similar messages [ 2902.724066] Lustre: server umount lustre-OST0001 complete [ 2914.295361] Lustre: server umount lustre-MDT0000 complete [ 2918.562252] LustreError: 100730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815530 with bad export cookie 945498313093970834 [ 2918.564497] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2918.575178] LustreError: 100730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2919.125063] Lustre: server umount lustre-MDT0001 complete [ 2928.475385] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 13:38:59 (1786815539) [ 2936.985098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2936.994382] Lustre: *** cfs_fail_loc=197, val=0*** [ 2952.543656] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2957.663859] LustreError: 103452:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.104@tcp: failed processing log, type 4: rc = -110 [ 2962.812594] Lustre: *** cfs_fail_loc=197, val=0*** [ 2962.817556] Lustre: Skipped 65 previous similar messages [ 2983.391467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2983.395548] Lustre: Skipped 4 previous similar messages [ 2990.463640] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 2998.923678] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3001.444204] Lustre: server umount lustre-OST0000 complete [ 3010.225475] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 13:40:21 (1786815621) [ 3030.931046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3036.222660] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3046.378354] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3050.708451] LustreError: 105103:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3050.718470] LustreError: 105103:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3050.725547] LustreError: 105103:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 3051.227888] Lustre: server umount lustre-MDT0000 complete [ 3085.610473] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 3096.497294] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 3138.430530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3138.811712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3138.853837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3138.966347] Lustre: lustre-MDT0000: new disk, initializing [ 3139.084598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3144.030751] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3156.796232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3156.912094] Lustre: 109280:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3157.121790] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3161.766142] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3171.204303] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3177.005273] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3184.247199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3184.479864] Lustre: 111251:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3190.280651] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3192.505235] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3198.886151] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3203.525200] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3203.742903] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3208.179397] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3208.415125] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3213.289346] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3213.295047] LustreError: Skipped 1 previous similar message [ 3213.295260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3213.299643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3213.299766] Lustre: Skipped 10 previous similar messages [ 3213.317068] Lustre: Skipped 6 previous similar messages [ 3217.009777] Lustre: server umount lustre-OST0000 complete [ 3228.273390] Lustre: server umount lustre-MDT0000 complete [ 3228.641712] LustreError: 109289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3228.657454] LustreError: 109289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3232.612672] LustreError: 109274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815844 with bad export cookie 945498313093981523 [ 3232.618573] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3232.640140] LustreError: 109274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3232.961410] Lustre: server umount lustre-MDT0001 complete [ 3241.436152] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 13:44:12 (1786815852) [ 3251.709154] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 3254.391473] Key type lgssc unregistered [ 3254.766676] LNet: 113741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3254.771926] LNetError: 113741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3254.789439] LNet: Removed LNI 192.168.201.104@tcp [ 3255.868466] Key type .llcrypt unregistered [ 3255.870346] Key type ._llcrypt unregistered [ 3273.523191] Key type ._llcrypt registered [ 3273.525365] Key type .llcrypt registered [ 3273.629147] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 3288.670121] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 3290.131419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3290.193766] alg: No test for adler32 (adler32-zlib) [ 3291.359465] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 3291.713143] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 3293.399226] Key type lgssc registered [ 3294.808751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3334.016661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3334.044720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3335.376482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3335.398317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3335.494568] Lustre: lustre-MDT0000: new disk, initializing [ 3335.565896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3335.583212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3340.632323] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3350.412097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3350.619494] Lustre: lustre-OST0000: new disk, initializing [ 3350.624190] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3350.630157] Lustre: Skipped 1 previous similar message [ 3350.635154] Lustre: 117912:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3350.695082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3356.352643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3356.359656] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3356.406972] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3356.755753] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3367.876390] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3368.117273] Lustre: lustre-OST0001: new disk, initializing [ 3368.135201] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3368.142878] Lustre: 118776:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3368.270845] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3374.113334] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3375.332717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3375.350891] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3375.384222] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3385.825957] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3385.833947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3385.847074] Lustre: Skipped 1 previous similar message [ 3385.872601] Lustre: Skipped 1 previous similar message [ 3390.902862] Lustre: server umount lustre-MDT0000 complete [ 3398.602292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3404.073331] LustreError: 117100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816016 with bad export cookie 11139334944784033640 [ 3404.088072] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3404.088614] LustreError: 117100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3404.185180] Lustre: server umount lustre-OST0000 complete [ 3408.346568] Lustre: server umount lustre-OST0001 complete [ 3418.441423] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 13:47:09 (1786816029) [ 3442.780604] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 13:47:33 (1786816053) [ 3452.796465] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 3473.111926] Lustre: DEBUG MARKER: == conf-sanity test 89a: check tunefs --param and --erase-param{s} options ========================================================== 13:48:04 (1786816084) [ 3510.696259] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 3519.600563] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 3555.302736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3555.614438] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3555.646625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3555.813523] Lustre: lustre-MDT0000: new disk, initializing [ 3555.916054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3555.931480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3560.790095] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3571.838224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3571.982633] Lustre: 126527:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3572.029044] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3572.033230] Lustre: Skipped 1 previous similar message [ 3572.094230] Lustre: lustre-MDT0001: new disk, initializing [ 3572.181801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3572.233233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3572.244622] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3577.243351] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3587.326102] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3593.562754] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3601.041688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3601.276317] Lustre: lustre-OST0000: new disk, initializing [ 3601.278933] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3601.286557] Lustre: 128495:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3601.373800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3608.097143] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3608.643788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3608.657951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3608.757806] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3616.933200] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3621.650849] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3621.832446] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3626.040688] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3626.206507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3629.537897] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3629.541979] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3629.543515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3629.568013] LustreError: Skipped 1 previous similar message [ 3634.684783] Lustre: server umount lustre-OST0000 complete [ 3639.781268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3639.789623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3639.807765] Lustre: Skipped 2 previous similar messages [ 3639.821891] Lustre: Skipped 2 previous similar messages [ 3644.900687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3644.905912] Lustre: Skipped 1 previous similar message [ 3645.502580] Lustre: server umount lustre-MDT0000 complete [ 3649.296680] LustreError: 126521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816261 with bad export cookie 11139334944784034865 [ 3649.297737] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3649.307932] LustreError: 126521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3649.687559] Lustre: server umount lustre-MDT0001 complete [ 3657.736320] Lustre: DEBUG MARKER: == conf-sanity test 89b: tunefs --quota keeps the other options and creates quota inodes ========================================================== 13:51:09 (1786816269) [ 3698.129628] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 3707.422756] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 3738.509154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3738.717143] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3738.736383] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3738.805456] Lustre: lustre-MDT0000: new disk, initializing [ 3738.864725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3738.873798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3743.069911] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3752.918304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3753.019475] Lustre: 134356:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3753.339760] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3757.688682] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3766.884381] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3772.780223] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3779.823804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3780.128541] Lustre: lustre-OST0000: new disk, initializing [ 3780.132283] Lustre: Skipped 1 previous similar message [ 3780.135446] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3780.140978] Lustre: Skipped 2 previous similar messages [ 3780.147300] Lustre: 136325:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3780.224705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3780.235201] Lustre: Skipped 1 previous similar message [ 3786.314244] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3787.805157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3787.817257] Lustre: Skipped 1 previous similar message [ 3787.826038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3787.942827] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3795.636729] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3800.563663] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3800.793585] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3806.223119] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3806.397726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3808.225444] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3808.231041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3808.241383] Lustre: Skipped 1 previous similar message [ 3808.242515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3808.253175] Lustre: Skipped 1 previous similar message [ 3814.278989] Lustre: server umount lustre-OST0000 complete [ 3818.466580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3818.481612] Lustre: Skipped 1 previous similar message [ 3818.484650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3818.484658] Lustre: Skipped 2 previous similar messages [ 3824.851721] Lustre: server umount lustre-MDT0000 complete [ 3828.461681] LustreError: 136324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816440 with bad export cookie 11139334944784035747 [ 3828.462409] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3828.471587] LustreError: 136324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3828.830398] Lustre: server umount lustre-MDT0001 complete [ 3832.563351] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 13:54:03 (1786816443) [ 3839.352104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3839.855851] LustreError: 138590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3839.872140] LustreError: 138590:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3844.787523] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3845.096773] LustreError: 138591:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3850.216582] LustreError: 138590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3853.414354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3853.823594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3853.837835] Lustre: Skipped 1 previous similar message [ 3858.209796] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3866.030225] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3871.906332] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3878.417575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3884.876829] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 3891.909660] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3906.118240] Lustre: *** cfs_fail_loc=159, val=0*** [ 3922.543452] Lustre: lustre-MDT0000: Client 9a779ef2-6f2b-42a7-9016-96a2f7c3bd09 (at 192.168.201.4@tcp) reconnecting [ 3924.222120] Lustre: *** cfs_fail_loc=159, val=0*** [ 3924.227458] Lustre: Skipped 5 previous similar messages [ 3940.514130] Lustre: lustre-MDT0000: Client 9a779ef2-6f2b-42a7-9016-96a2f7c3bd09 (at 192.168.201.4@tcp) reconnecting [ 3946.468890] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3946.477669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3946.501681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3946.504064] Lustre: Skipped 3 previous similar messages [ 3947.102834] Lustre: server umount lustre-OST0000 complete [ 3951.073373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3957.072924] Lustre: server umount lustre-MDT0000 complete [ 3959.777477] LustreError: 141274:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3959.797652] LustreError: 141274:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3961.059178] LustreError: 138572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816573 with bad export cookie 11139334944784036629 [ 3961.063561] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3961.068979] LustreError: 138572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3971.717996] Lustre: server umount lustre-MDT0001 complete [ 3983.018301] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 3985.742301] Key type lgssc unregistered [ 3985.987197] LNet: 142397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3985.992581] LNetError: 142397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3986.009704] LNet: Removed LNI 192.168.201.104@tcp [ 3986.854758] Key type .llcrypt unregistered [ 3986.859534] Key type ._llcrypt unregistered [ 4011.703867] Key type ._llcrypt registered [ 4011.706960] Key type .llcrypt registered [ 4011.887976] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 4013.442273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4013.476556] alg: No test for adler32 (adler32-zlib) [ 4014.569903] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 4014.957580] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 4016.687613] Key type lgssc registered [ 4017.840196] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4025.996693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4026.034711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4027.451186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4032.043754] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4037.603484] LustreError: 143812:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4037.618465] LustreError: 143812:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4040.074813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4040.547945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4045.015637] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4053.384840] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4058.704513] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4065.292195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4065.819370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4071.008537] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4072.002925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4078.594243] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4090.970816] Lustre: *** cfs_fail_loc=159, val=0*** [ 4106.342525] Lustre: lustre-MDT0000: Client 5210fa85-83a7-45b7-baed-666ac0fe2665 (at 192.168.201.4@tcp) reconnecting [ 4108.250723] Lustre: *** cfs_fail_loc=159, val=0*** [ 4124.833294] Lustre: lustre-MDT0001: Client 5210fa85-83a7-45b7-baed-666ac0fe2665 (at 192.168.201.4@tcp) reconnecting [ 4126.596384] Lustre: *** cfs_fail_loc=159, val=0*** [ 4126.599299] Lustre: Skipped 3 previous similar messages [ 4142.229713] Lustre: lustre-MDT0001: Client 5210fa85-83a7-45b7-baed-666ac0fe2665 (at 192.168.201.4@tcp) reconnecting [ 4158.853983] Lustre: 146871:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4161.469596] Lustre: *** cfs_fail_loc=159, val=0*** [ 4161.475630] Lustre: Skipped 4 previous similar messages [ 4178.073123] Lustre: lustre-MDT0000: Client 4d54498a-1d6f-42e7-af74-559838f550ee (at 192.168.201.4@tcp) reconnecting [ 4180.841940] Lustre: *** cfs_fail_loc=159, val=0*** [ 4180.865145] Lustre: Skipped 16 previous similar messages [ 4196.484180] Lustre: lustre-MDT0000: Client 4d54498a-1d6f-42e7-af74-559838f550ee (at 192.168.201.4@tcp) reconnecting [ 4198.313670] Lustre: 147163:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4202.050718] LustreError: 145633:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f3a36573 x1873613031551104/t0(0) o104->lustre-OST0000@192.168.201.4@tcp:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 [ 4204.013891] LustreError: lustre-OST0000-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 4204.016532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4204.035182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4205.024325] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4205.027184] LustreError: Skipped 2 previous similar messages [ 4205.031154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4205.042047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4208.105867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4208.110707] Lustre: Skipped 1 previous similar message [ 4210.157876] Lustre: server umount lustre-OST0000 complete [ 4214.754279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4214.764431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4214.780327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4218.341221] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4219.909771] Lustre: server umount lustre-MDT0000 complete [ 4223.290216] LustreError: 146992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816835 with bad export cookie 11489124162665241755 [ 4223.298605] LustreError: 146992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4223.303749] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4223.708454] Lustre: server umount lustre-MDT0001 complete [ 4235.705702] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 4237.785660] Key type lgssc unregistered [ 4238.063192] LNet: 148226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4238.070157] LNetError: 148226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4238.086558] LNet: Removed LNI 192.168.201.104@tcp [ 4239.179056] Key type .llcrypt unregistered [ 4239.182332] Key type ._llcrypt unregistered [ 4264.072969] Key type ._llcrypt registered [ 4264.075855] Key type .llcrypt registered [ 4264.174587] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 4265.577048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4265.669150] alg: No test for adler32 (adler32-zlib) [ 4266.762578] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 4266.926403] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 4268.569177] Key type lgssc registered [ 4269.543587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4278.808333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4278.843201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4280.242352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4285.367713] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4290.529636] LustreError: 149652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4290.541675] LustreError: 149652:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4293.765550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4294.135562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4299.634630] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4308.074572] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4314.492735] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4321.571827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4322.104534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4325.229347] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4329.455513] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4337.531543] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4339.533777] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4346.429121] Lustre: 152235:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4349.162029] Lustre: 152283:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4350.595155] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4351.725518] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4355.283473] Lustre: 152429:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4356.252074] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4357.183973] Lustre: 152528:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4365.286548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4365.303986] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4365.334560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4365.355692] Lustre: Skipped 1 previous similar message [ 4367.683805] Lustre: server umount lustre-OST0000 complete [ 4375.521231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4375.526777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4375.544291] Lustre: Skipped 2 previous similar messages [ 4375.557212] Lustre: Skipped 1 previous similar message [ 4380.648067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4380.657835] Lustre: Skipped 1 previous similar message [ 4385.765986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4385.775653] Lustre: Skipped 1 previous similar message [ 4386.271763] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4386.597537] Lustre: server umount lustre-MDT0000 complete [ 4390.881787] LustreError: 149652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4390.921707] LustreError: 149652:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4391.300749] LustreError: 149633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786817003 with bad export cookie 5843324919701477659 [ 4391.322356] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4391.329280] LustreError: 149633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4391.895901] Lustre: server umount lustre-MDT0001 complete [ 4406.986551] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 4409.686976] Key type lgssc unregistered [ 4409.949294] LNet: 153596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4409.954283] LNetError: 153596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4409.967796] LNet: Removed LNI 192.168.201.104@tcp [ 4411.031461] Key type .llcrypt unregistered [ 4411.034976] Key type ._llcrypt unregistered [ 4439.554852] Key type ._llcrypt registered [ 4439.557906] Key type .llcrypt registered [ 4439.669080] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 4441.029196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4441.196717] alg: No test for adler32 (adler32-zlib) [ 4442.391641] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 4442.779641] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 4444.456102] Key type lgssc registered [ 4445.922716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4454.312416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4454.328885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4455.916435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4461.775592] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4465.125188] LustreError: 155013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4465.162171] LustreError: 155013:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4470.246764] LustreError: 155012:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4471.209297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4471.814349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4477.070603] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4485.704464] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4492.421819] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4499.896334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4500.553273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4505.590945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4508.383224] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4516.827183] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4516.860558] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4522.529209] Lustre: *** cfs_fail_loc=159, val=0*** [ 4539.069974] Lustre: lustre-MDT0000: Client e204b91b-6708-42d6-bbf5-80e656f2c796 (at 192.168.201.4@tcp) reconnecting [ 4556.793381] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4556.804049] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4556.831856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4557.794487] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4557.798925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4557.799787] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4562.116361] Lustre: server umount lustre-OST0000 complete [ 4568.044829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4568.068053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4568.071888] Lustre: Skipped 1 previous similar message [ 4568.085475] Lustre: Skipped 2 previous similar messages [ 4572.829570] Lustre: server umount lustre-MDT0000 complete [ 4573.152265] LustreError: 155007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4573.162477] LustreError: 155007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4576.899291] LustreError: 157628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786817189 with bad export cookie 8212825456448353264 [ 4576.908935] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4576.912240] LustreError: 157628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4577.176419] Lustre: server umount lustre-MDT0001 complete [ 4589.861884] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 4593.106921] Key type lgssc unregistered [ 4593.406796] LNet: 158686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4593.414627] LNetError: 158686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4593.433774] LNet: Removed LNI 192.168.201.104@tcp [ 4594.319156] Key type .llcrypt unregistered [ 4594.320910] Key type ._llcrypt unregistered [ 4620.853343] Key type ._llcrypt registered [ 4620.856339] Key type .llcrypt registered [ 4621.073452] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 4622.661310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4622.688525] alg: No test for adler32 (adler32-zlib) [ 4623.760496] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 4624.085685] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 4625.759321] Key type lgssc registered [ 4627.127958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4635.922516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4635.948539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4637.460300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4642.643430] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4646.889960] LustreError: 160101:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4646.915659] LustreError: 160101:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4652.003107] LustreError: 160102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4652.392060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4652.937214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4657.242125] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4668.138155] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4674.407568] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4682.291835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4682.695924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4683.725867] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4689.507289] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4697.026466] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4697.609632] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4717.159618] Lustre: lustre-OST0000: evicting a853090d-6204-4721-b7bd-73164c748044 (at 192.168.201.4@tcp) by administrative request [ 4717.166798] Lustre: Skipped 1 previous similar message [ 4748.257118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4748.269705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4748.294445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4748.303234] Lustre: Skipped 1 previous similar message [ 4752.688324] Lustre: server umount lustre-OST0000 complete [ 4758.498726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4758.500919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4758.521829] Lustre: Skipped 1 previous similar message [ 4758.545154] Lustre: Skipped 2 previous similar messages [ 4763.060981] Lustre: server umount lustre-MDT0000 complete [ 4763.617899] LustreError: 160832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4763.641754] LustreError: 160832:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4767.118515] LustreError: 160081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786817379 with bad export cookie 10352848424225488141 [ 4767.125952] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4767.128868] LustreError: 160081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4767.692808] Lustre: server umount lustre-MDT0001 complete [ 4781.706558] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 4785.423855] Key type lgssc unregistered [ 4785.773552] LNet: 164178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4785.780610] LNetError: 164178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4785.797429] LNet: Removed LNI 192.168.201.104@tcp [ 4786.864870] Key type .llcrypt unregistered [ 4786.867530] Key type ._llcrypt unregistered [ 4864.122294] Key type ._llcrypt registered [ 4864.132843] Key type .llcrypt registered [ 4864.342937] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 4867.173819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4867.300453] alg: No test for adler32 (adler32-zlib) [ 4868.390870] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 4868.747886] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 4870.447193] Key type lgssc registered [ 4872.191151] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4879.206853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4879.260468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4880.688360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4884.460804] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4890.083413] LustreError: 166680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4890.115962] LustreError: 166680:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4893.355651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4893.713823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4898.621942] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4906.339756] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4912.523581] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4920.146359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4920.569191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4925.698834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4927.711876] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 4935.336467] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4937.979565] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4946.401132] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4946.413549] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4946.428472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4948.452089] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4948.461573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4948.468291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4948.494039] Lustre: Skipped 1 previous similar message [ 4950.706855] Lustre: server umount lustre-OST0000 complete [ 4955.104133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4955.110143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4955.125916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4958.688646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4958.691194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4960.964594] Lustre: server umount lustre-MDT0000 complete [ 4963.809944] LustreError: 167681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4963.842962] LustreError: 167681:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4965.118913] LustreError: 167423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786817577 with bad export cookie 14710147243218382658 [ 4965.128308] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4965.128550] LustreError: 167423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4965.652701] Lustre: server umount lustre-MDT0001 complete [ 4978.685085] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 4981.976843] Key type lgssc unregistered [ 4982.328861] LNet: 170225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4982.354709] LNetError: 170225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4982.386638] LNet: Removed LNI 192.168.201.104@tcp [ 4984.107859] Key type .llcrypt unregistered [ 4984.110212] Key type ._llcrypt unregistered [ 5011.399592] Key type ._llcrypt registered [ 5011.401784] Key type .llcrypt registered [ 5011.536895] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 5012.568864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5012.586453] alg: No test for adler32 (adler32-zlib) [ 5013.744778] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 5014.067689] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 5015.775158] Key type lgssc registered [ 5017.135290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5033.359223] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 14:14:04 (1786817644) [ 5049.897290] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 5063.176575] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 5104.879708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5104.934941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5106.477407] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5106.503291] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5106.660167] Lustre: lustre-MDT0000: new disk, initializing [ 5106.805456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5106.854720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5113.052369] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5125.591052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5125.665236] Lustre: 174493:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5125.715090] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5125.718185] Lustre: Skipped 1 previous similar message [ 5125.819521] Lustre: lustre-MDT0001: new disk, initializing [ 5126.083585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5126.167796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5126.183211] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5131.680153] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5142.229511] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5149.013626] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5156.614574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5156.920843] Lustre: lustre-OST0000: new disk, initializing [ 5156.927887] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5156.938968] Lustre: 176464:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5157.089910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5162.594942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5162.612900] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5162.682053] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5164.282133] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5174.038407] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5188.587261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5188.603518] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5188.603910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5188.606534] Lustre: Skipped 1 previous similar message [ 5189.927490] Lustre: server umount lustre-OST0000 complete [ 5197.801567] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5197.818288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5197.850757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5197.865772] Lustre: Skipped 1 previous similar message [ 5198.819596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5200.323961] Lustre: server umount lustre-MDT0000 complete [ 5203.896545] LustreError: 174484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786817816 with bad export cookie 9444146166130629708 [ 5203.897878] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5203.910818] LustreError: 174484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5204.276849] Lustre: server umount lustre-MDT0001 complete [ 5216.118625] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 5219.103975] Key type lgssc unregistered [ 5219.499935] LNet: 178372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5219.521692] LNetError: 178372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5219.568898] LNet: Removed LNI 192.168.201.104@tcp [ 5220.687670] Key type .llcrypt unregistered [ 5220.689705] Key type ._llcrypt unregistered [ 5246.549367] Key type ._llcrypt registered [ 5246.551825] Key type .llcrypt registered [ 5246.721432] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 5248.035254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5248.181822] alg: No test for adler32 (adler32-zlib) [ 5249.311860] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 5249.504886] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 5251.191176] Key type lgssc registered [ 5252.382755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5259.869764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5259.916806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5261.400212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5264.670102] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5271.525791] LustreError: 179784:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5271.540916] LustreError: 179784:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5273.407386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5273.783459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5278.789940] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5286.928971] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5292.977158] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5299.527698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5300.088463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5306.233491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5307.361704] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 5314.591208] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5324.115296] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5324.145390] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5324.167608] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5324.183086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.200693] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5325.201438] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5325.216894] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5325.223820] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5325.233365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5325.249833] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5326.158217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.909266] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5326.912495] Lustre: Skipped 1 previous similar message [ 5326.930416] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5326.943909] Lustre: Skipped 1 previous similar message [ 5326.947903] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5326.953169] Lustre: Skipped 1 previous similar message [ 5326.955884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.964026] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5326.970526] Lustre: Skipped 1 previous similar message [ 5327.799803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.591414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.625632] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5329.630489] Lustre: Skipped 2 previous similar messages [ 5329.654301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5329.663692] Lustre: Skipped 2 previous similar messages [ 5329.669525] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5329.672029] Lustre: Skipped 2 previous similar messages [ 5329.676908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.686522] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5329.696709] Lustre: Skipped 2 previous similar messages [ 5330.564654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.498688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.312433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.417266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.267713] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5334.273553] Lustre: Skipped 4 previous similar messages [ 5334.283882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5334.295571] Lustre: Skipped 4 previous similar messages [ 5334.305475] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5334.308839] Lustre: Skipped 4 previous similar messages [ 5334.318476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.336289] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5334.343159] Lustre: Skipped 4 previous similar messages [ 5335.032887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.986146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.017420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.896447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.804071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.825362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.915564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.846179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.807204] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5342.815760] Lustre: Skipped 8 previous similar messages [ 5342.847832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5342.857227] Lustre: Skipped 8 previous similar messages [ 5342.861761] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5342.864419] Lustre: Skipped 8 previous similar messages [ 5342.870701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.880700] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5342.900758] Lustre: Skipped 8 previous similar messages [ 5343.784253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.818351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.875660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.924466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.837987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.920547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.968418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.852382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.651649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.823719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.708266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.603878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.648849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.750562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.898419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.818339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.825884] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5359.831430] Lustre: Skipped 16 previous similar messages [ 5359.842795] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5359.851945] Lustre: Skipped 16 previous similar messages [ 5359.856144] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5359.859900] Lustre: Skipped 16 previous similar messages [ 5359.861957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.884423] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5359.903726] Lustre: Skipped 16 previous similar messages [ 5360.776499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.912464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.805633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.834422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.683655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.765215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.618751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.633341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.433714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.399285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.442971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.514559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.420487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.386906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.501317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.414391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.366271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.315070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.399948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.088142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.189708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.077253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.989266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.724972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.758820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.623163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.320237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.356935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.240592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.265482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.991232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.839324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.883677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.761739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.616238] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5392.621291] Lustre: Skipped 34 previous similar messages [ 5392.635156] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5392.642944] Lustre: Skipped 34 previous similar messages [ 5392.646560] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5392.651656] Lustre: Skipped 34 previous similar messages [ 5392.653804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.660325] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5392.662438] Lustre: Skipped 34 previous similar messages [ 5393.412961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.575034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.524138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.681954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.785144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.003756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.123989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.161379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.112788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.026799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.976484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.976803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.861112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.838957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.842434] LustreError: 179804:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5407.852152] LustreError: 179804:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5407.857240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.807653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.907749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.859772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.098088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.113758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.157594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.258665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.157961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.338797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.152912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.868298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.849190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.948411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.788173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.632299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.561547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.668044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.663702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.739740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.886951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.969629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.135578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.255495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.317544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.392621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.423805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.371343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.413431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.214532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.189593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.017584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.058917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.950194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.031354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.857349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.813239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.995879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.166734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.117353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.157692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.342037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.089552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.742593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.677805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.865184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.890079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.024468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.140045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.409260] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5457.411977] Lustre: Skipped 63 previous similar messages [ 5457.425077] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5457.429615] Lustre: Skipped 63 previous similar messages [ 5457.431556] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5457.434102] Lustre: Skipped 63 previous similar messages [ 5457.436543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.444725] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5457.446796] Lustre: Skipped 63 previous similar messages [ 5458.985121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.281308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.632636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.871883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.295321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.604590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.917199] LustreError: 179804:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5466.932737] LustreError: 179804:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5466.939867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.894280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.215579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.690435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.112926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.248297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.541910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.854258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.976918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.772365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.010393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.059603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.251107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.963647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.887580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.649359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.429317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.468956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.469900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.495907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.234929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.378584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.439949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.152715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.913179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.777512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.062289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.095751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.185271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.283031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.053307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.073689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.066377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.193522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.950768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.040435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.940738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.181987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.160234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.301427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.238618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.216437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.066482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.137164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.086151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.028934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.106515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.896675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.000537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.988841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.969981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.966255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.822172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.824676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.665160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.668653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.633965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.667768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.653385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.675551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.581305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.585208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.411346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.514286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.577315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.426094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.320227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.332872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.501071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.769506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.966118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.279978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.373618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.516468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.491722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.717453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.621244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.553473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.412793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.296414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.165841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.000129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.818517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.616374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.494957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.295547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.169241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.032282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.860981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.809804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.867640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.844157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.674744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.529473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.482072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.541284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.440756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.325531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.154298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.166583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.146285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.108684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.885807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.831637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.561844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.497159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.516092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.476134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.391118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.368336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.233625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.075892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.919646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.685458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.414756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.180588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.893297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.789906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.681797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.514875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.224506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.117096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.802535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.697784] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5585.701562] Lustre: Skipped 129 previous similar messages [ 5585.714779] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5585.721786] Lustre: Skipped 129 previous similar messages [ 5585.725818] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5585.729510] Lustre: Skipped 129 previous similar messages [ 5585.732735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.739625] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5585.742314] Lustre: Skipped 129 previous similar messages [ 5586.578815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.738948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.797301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.951874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.788772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.806035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.569467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.473279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.174368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.980580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.811480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.836606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.671463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.675694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.339364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.327573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.231171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.310385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.240887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.157549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.945530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.798831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.568957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.360209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.086703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.972673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.776566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.676590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.571423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.511060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.288257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.205248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.976347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.841203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.658776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.638546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.406974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.248957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.983599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.740972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.427597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.217457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.999739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.874622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.657078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.471825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.298780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.168317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.017499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.810977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.802310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.633176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.461371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.236614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.247145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.025682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.879341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.810589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.736568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.592422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.512790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.471591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.577357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.395532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.379857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.282066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.358849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.136658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.948899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.686345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.517718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.216338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.162536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.068647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.902933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.841246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.989315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.933906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.907748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.942370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.930928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.776519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.704961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.469680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.296220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.159253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.957853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.612280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.543927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.456776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.428586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.408406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.470922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.377895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.636285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.827267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.796628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.998316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.263864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.281110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.317921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.273654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.332519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.362263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.381079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.356325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.442212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.232106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.946934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.720996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.698429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.647577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.571517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.330970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.132026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.001085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.918883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.740256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.442933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.234343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.941438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.734750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.681455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.605789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.439350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.002040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.891101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.724748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.513886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.399653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.141655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.127720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.804182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.831550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.612438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.503418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.391585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.154900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.044065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.857405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.841748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.774324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.774495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.671320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.511168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.320746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.242818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.152993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.958166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.713445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.774808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.682121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.452293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.567926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.392794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.373856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.338114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.372430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.357257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.377899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.298916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.293335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.134680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.997606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.819630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.741454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.640248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.624952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.693591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.701785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.999321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.163489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.266113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.588603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.744455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.967492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.146198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.301699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.549576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.980490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.169628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.238433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.521647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.729313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.902418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.161689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.084916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.940953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.866986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.828608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.807141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.653396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.420482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.597251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.459167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.533498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.596569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.532189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.265669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.914876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.082473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.167526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.236082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.049187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.917339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.698902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.961719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.924265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.832173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.484327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.384966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.167935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.023731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.905907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.122444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.252680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.098455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.356921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.690642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.939980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.900399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.138275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.117262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.096605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.022368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.026319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.997792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.941375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.706450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.544251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.243031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.109755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.745501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.681566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.384185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.305233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.529561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.683267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.691234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.614775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.484853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.400195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.233168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.251568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.098367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.991885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.914710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.008027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.989735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.987534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.780063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.674467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.582720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.628460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.276110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.191526] LustreError: 179804:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5823.207479] LustreError: 179804:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5823.211435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.307782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.216481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.152668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.070645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.034845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.934828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.712174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.698426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.571150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.424667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.406837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.226709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.106828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.020330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.090596] LustreError: 179804:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5837.104854] LustreError: 179804:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5837.125297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.122858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.188723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.259627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.210588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.369855] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5842.380560] Lustre: Skipped 275 previous similar messages [ 5842.403813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5842.419226] Lustre: Skipped 275 previous similar messages [ 5842.424917] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5842.430474] Lustre: Skipped 275 previous similar messages [ 5842.433780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.444178] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5842.449816] Lustre: Skipped 275 previous similar messages [ 5843.562697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.326401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.343708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.173803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.358791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.126555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.370449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.326389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.453152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.419555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.515967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.501317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.732900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.062629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.238305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.244365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.131776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.213265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.375182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.381084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.474575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.387222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.495425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.656986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.615683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.867767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.068890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.035956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.213464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.054651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.046854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.959510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.915717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.052250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.096351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.298933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.430920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.454658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.443452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.308237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.186589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.158526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.248894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.185967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.238719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.395573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.382258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.212213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.067105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.154483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.099361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.274678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.259889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.071903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.863134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.665566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.891630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.156891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.334071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.602662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.479102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.642332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.800396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.020188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.993172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.174581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.334063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.303988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.314567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.120066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.125534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.086312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.095495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.119552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.077209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.057905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.098065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.954873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.826150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.560959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.478213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.484397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.878270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.891300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.953476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.365803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.788551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.422717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.892123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.106301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.613583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.917042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.989583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.473844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.405856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.771643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.113027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.892528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.778795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.423258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.925194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.849713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.348533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.220504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.037745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.154358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.817131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.619943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.276949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.113684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.972473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.056211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.943603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.072529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.570777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.035147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.359251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.456672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.725815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.367414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.205547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.138677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.067866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.208842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.290497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.252951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.408631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.754712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.104490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.078169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.081319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.997145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.799508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.731239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.702041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.833930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.836370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.900497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.595359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.703244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.782268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.027816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.861517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.039584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.941904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.759285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.918548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.653675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.610415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.481169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.492225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.554869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.690226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.459431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.297554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.024243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.910132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.996518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.965874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.043570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.223925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.272294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.260887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.322497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.196581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.177643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.252545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.345585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.182694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.048741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.223938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.434090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.386740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.216686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.003773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.863618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.828138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.986947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.998148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.982834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.985461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.020162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.163481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.147684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.169982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.155905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.236103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.087630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.058526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.214860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.172690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.068905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.989456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.025866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.923446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.809331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.809595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.803848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.942250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.899105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.885046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.947546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.894872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.962390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.837276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.729721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.624192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.673214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.492694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.630469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.739976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.775753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.953475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.879440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.947213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.860362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.776386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.518676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.488938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.368461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.507061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.382643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.336550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.215192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.141513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.013354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.048875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.940627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.882423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.696097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.779915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.675129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.628982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.817824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.616113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.637838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.562067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.591883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.647865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.541757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.585491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.676530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.656293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.707996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.776424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.933790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.949994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.902798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.758629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.688451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.623247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.749749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.668053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.689947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.679892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.855764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.887421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.934969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.972883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.990645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.144462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.316915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.479315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.258534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.334439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.314834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.302657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.471863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.589629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.406715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.503069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.643818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.633071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.579370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.635886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.481415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.570219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.778879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.813694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.767510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.879962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.246945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.317730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.278797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.269716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.297594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.975520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.820288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.745320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.731282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.499681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.496929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6140.360862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.141450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.868352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.635350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.486373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.405960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.365945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.161641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.057809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.985065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.710112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.778617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6150.718309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.637887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.372833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6153.329390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6154.147048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.142191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.036361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6157.031052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6157.815205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6158.685933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6159.828953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6160.951565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.992950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.914253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6163.767890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6164.921502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.836432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.806225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6167.793512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6168.835422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6169.829193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6171.047289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6172.216385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6173.227045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.251607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6175.266581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6176.372935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.141461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6178.123430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6179.089689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6180.155755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6181.148271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6182.172577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.126664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.207123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6185.237524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6186.077799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.166887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.473247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6189.548837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.596272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.562850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6192.737620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6193.676468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.533577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.574514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.276377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.080139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.963309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6199.039912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6199.940561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6200.979625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6201.791343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6202.837928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.760063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6205.057786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.172900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6207.143617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6208.701513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6209.718065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6210.781791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6211.863506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6213.076739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6214.237331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6215.045522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6215.935749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6217.011084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6218.141863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6219.074271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6220.091917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6221.181224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6222.151303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6223.337985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6224.191358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6225.248622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6226.433902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6227.374846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6228.467631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6229.356360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6230.093602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6231.066116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6231.882280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6233.035801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6233.961644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6235.043963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6236.170785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6237.162295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6238.267311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6239.307517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6240.289678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6241.372705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6242.410403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6243.512751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6244.470930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6245.427032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6246.272234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6247.289443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6248.093684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6248.968444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6250.093835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6251.159157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6252.031948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6252.782228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6253.852277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6254.805781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6255.521869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6256.465321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6257.410572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6258.493255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6259.322470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6260.613168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6261.556134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6262.554983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6263.492945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6264.835031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6265.968307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6267.031768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6268.079775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6269.149619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6270.397615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6270.987203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6271.869178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6272.787808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6274.394059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6275.351802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6276.332419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6277.286549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6278.117470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6278.937965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6280.054115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6280.961872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6282.046793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6283.115581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6284.038718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6285.060373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6286.067134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6287.062203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6288.123499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6289.654588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6290.531542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6291.383264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6292.260313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6293.496350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6295.007768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6295.816891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6296.898532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6297.925066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6299.097149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6300.014732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6301.239099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6302.515619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6303.592800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6305.099817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6633.705501] LustreError: 196826:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6633.723753] LustreError: 179804:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 7105.508077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7105.511769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7105.527071] Lustre: Skipped 458 previous similar messages [ 7110.635900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7110.646047] Lustre: Skipped 2 previous similar messages [ 7110.966710] Lustre: server umount lustre-OST0000 complete [ 7115.751716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7115.758764] Lustre: Skipped 1 previous similar message [ 7120.871066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7120.876559] Lustre: Skipped 1 previous similar message [ 7122.383851] Lustre: server umount lustre-MDT0000 complete [ 7125.985949] LustreError: 179784:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7126.010838] LustreError: 179784:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7126.672844] LustreError: 179764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819738 with bad export cookie 11063830915579084047 [ 7126.689553] LustreError: 179764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7126.691237] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7127.342790] Lustre: server umount lustre-MDT0001 complete [ 7141.433920] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 7144.632824] Key type lgssc unregistered [ 7144.977257] LNet: 235006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7144.983795] LNetError: 235006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7144.995738] LNet: Removed LNI 192.168.201.104@tcp [ 7146.100605] Key type .llcrypt unregistered [ 7146.102285] Key type ._llcrypt unregistered [ 7173.020507] Key type ._llcrypt registered [ 7173.022281] Key type .llcrypt registered [ 7173.106918] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 7174.629712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7174.646341] alg: No test for adler32 (adler32-zlib) [ 7175.675671] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 7175.941187] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 7177.591177] Key type lgssc registered [ 7178.702882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7186.727273] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7186.738584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7188.087843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7191.701789] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7198.696977] LustreError: 236422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7198.715295] LustreError: 236422:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7199.717295] LustreError: 236421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7200.809152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7201.365299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7206.011842] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7213.913166] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7219.827605] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7226.361348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7226.862707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7233.397579] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7234.047514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7242.914372] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7248.265452] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7250.475552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7250.485112] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7250.501806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7250.535850] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7250.539352] LustreError: 236442:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7253.474423] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7253.488802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7253.508151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7254.498464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7254.503688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7254.532738] Lustre: Skipped 1 previous similar message [ 7256.180052] Lustre: lustre-OST0000: Not available for connect from 192.168.201.4@tcp (stopping) [ 7258.888982] Lustre: server umount lustre-OST0000 complete [ 7270.617162] LustreError: 236442:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7292.240876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7292.493572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7294.583752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7294.590465] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7298.567872] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7304.822306] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7304.836555] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7305.530191] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7307.284209] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7312.242982] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8d3d1244e000.ost_server_uuid 50 [ 7313.692632] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8d3d1244e000.ost_server_uuid in FULL state after 0 sec [ 7319.015908] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7319.034150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7319.072654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7320.551905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7324.002609] Lustre: server umount lustre-OST0000 complete [ 7329.248587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7329.257387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7329.276697] Lustre: Skipped 1 previous similar message [ 7329.295431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7329.314920] Lustre: Skipped 2 previous similar messages [ 7340.022241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7340.031664] Lustre: Skipped 5 previous similar messages [ 7342.047168] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7342.320951] Lustre: server umount lustre-MDT0000 complete [ 7343.075127] LustreError: 237149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7343.084962] LustreError: 237149:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7345.124602] LustreError: 236416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7345.144827] LustreError: 236416:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7346.514398] LustreError: 236403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819958 with bad export cookie 6532809688633659943 [ 7346.518530] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7346.519906] LustreError: 236403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7346.933252] Lustre: server umount lustre-MDT0001 complete [ 7358.468233] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 7361.624363] Key type lgssc unregistered [ 7362.055180] LNet: 241632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7362.070911] LNetError: 241632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7362.115786] LNet: Removed LNI 192.168.201.104@tcp [ 7362.930255] Key type .llcrypt unregistered [ 7362.932806] Key type ._llcrypt unregistered [ 7392.193487] Key type ._llcrypt registered [ 7392.195500] Key type .llcrypt registered [ 7392.313613] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 7393.325451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7393.502243] alg: No test for adler32 (adler32-zlib) [ 7394.490629] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 7394.751228] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 7396.487168] Key type lgssc registered [ 7397.638415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7407.211741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7407.226175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7408.680107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7412.626731] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7418.849249] LustreError: 243362:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7418.876119] LustreError: 243362:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7420.990830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7421.487672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7426.161272] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7429.056174] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7435.904616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7436.268956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7443.124835] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7452.781859] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7453.067852] Lustre: lustre-OST0001: new disk, initializing [ 7453.076829] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7453.085844] Lustre: 245672:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7453.274634] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7454.356268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7460.548070] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7461.010026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7461.013877] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7461.126749] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7472.376194] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7480.393152] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7497.188957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7497.196387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7497.212228] Lustre: Skipped 3 previous similar messages [ 7497.233776] Lustre: Skipped 3 previous similar messages [ 7502.310784] LustreError: 244616:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7502.332845] LustreError: 244616:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7502.529303] Lustre: server umount lustre-MDT0000 complete [ 7506.433527] LustreError: 246441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820118 with bad export cookie 15333903815572795297 [ 7506.434926] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7506.440990] LustreError: 246441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7506.899151] Lustre: server umount lustre-MDT0001 complete [ 7521.574533] Lustre: server umount lustre-OST0000 complete [ 7523.808656] Lustre: 242625:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786820119/real 1786820119] req@ffff9b6d47143480 x1873616574711552/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786820135 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7523.861063] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7526.520426] Lustre: server umount lustre-OST0001 complete [ 7549.489811] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 7559.859520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7560.155215] LustreError: 248782:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7560.171841] LustreError: 248782:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7560.246752] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7564.368563] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7565.291524] LustreError: 248783:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7570.405391] LustreError: 248782:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7573.408415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7573.886565] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7578.474093] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7581.096619] Lustre: 249922:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7587.861620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7597.224652] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7598.377477] LustreError: 250277:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7598.388157] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:100 to 0x280000400:129) [ 7605.841154] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49760) [ 7606.775328] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7607.124891] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7607.127537] Lustre: Skipped 1 previous similar message [ 7614.340711] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7622.522795] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7626.249333] Lustre: 251792:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7635.431540] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7635.440552] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7635.468224] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7635.893943] Lustre: server umount mylustre-MDT0000 complete [ 7637.987839] Lustre: mylustre-MDT0000-lwp-OST0000: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7637.997343] LustreError: 248782:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7638.011746] Lustre: Skipped 2 previous similar messages [ 7638.049611] LustreError: 248782:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 7639.981591] LustreError: 248757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820252 with bad export cookie 15333903815572798160 [ 7639.992578] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7639.994508] LustreError: 248757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7642.442841] Lustre: server umount mylustre-MDT0001 complete [ 7655.112589] Lustre: server umount mylustre-OST0000 complete [ 7659.049167] Lustre: 242624:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786820255/real 1786820255] req@ffff9b6d471c5500 x1873616574773888/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786820271 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7659.075264] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7659.303695] Lustre: server umount mylustre-OST0001 complete [ 7680.361043] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 7690.137531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7690.642598] LustreError: 254102:0:(ldlm_lib.c:1192:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7690.777703] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7694.464190] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7702.934382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7707.732782] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7710.570451] Lustre: 255244:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7718.008765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7724.344048] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7728.650766] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7734.615931] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7735.930028] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49792) [ 7742.578827] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7750.146831] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7754.390545] Lustre: 257118:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7764.962698] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7764.975238] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7764.987528] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 7767.030304] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 7769.289748] Lustre: server umount tfs-MDT0000 complete [ 7772.138589] LustreError: 255499:0:(ldlm_lib.c:1192:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7772.173585] LustreError: 255499:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 7773.224187] LustreError: 254079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820385 with bad export cookie 15333903815572800113 [ 7773.229596] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7773.241922] LustreError: 254079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7773.650549] Lustre: server umount tfs-MDT0001 complete [ 7787.235929] Lustre: server umount tfs-OST0000 complete [ 7800.185866] Lustre: server umount tfs-OST0001 complete [ 7821.432734] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 7831.939369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7832.355950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7832.365896] Lustre: Skipped 3 previous similar messages [ 7837.772236] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7847.891716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7854.164888] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7857.580787] Lustre: 260521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7864.866623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7871.449619] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7873.669038] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 7877.746230] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49824) [ 7881.980849] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7889.897142] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 7898.378536] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7902.119261] Lustre: 262392:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7909.816151] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 15:02:01 (1786820521) [ 7925.230974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7925.251077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7925.280225] Lustre: Skipped 3 previous similar messages [ 7925.295089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7925.297876] Lustre: Skipped 3 previous similar messages [ 7931.775730] Lustre: server umount lustre-MDT0000 complete [ 7933.417246] LustreError: 259364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7933.427287] LustreError: 259364:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 7935.431587] LustreError: 259345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820547 with bad export cookie 15333903815572801807 [ 7935.438118] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7935.444561] LustreError: 259345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7938.536048] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7938.549043] Lustre: Skipped 4 previous similar messages [ 7942.086388] Lustre: server umount lustre-MDT0001 complete [ 7952.419616] Lustre: server umount lustre-OST0000 complete [ 7962.718916] Lustre: server umount lustre-OST0001 complete [ 7968.536250] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 7978.826453] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 8021.721466] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 8021.721466] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8021.721466] [ 8021.792090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8022.361211] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8022.393981] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8022.559617] Lustre: lustre-MDT0000: new disk, initializing [ 8022.742028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8022.750484] Lustre: Skipped 3 previous similar messages [ 8022.810665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8028.069588] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8039.430778] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 8039.430778] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8039.430778] [ 8039.472989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8039.658237] Lustre: 265576:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8039.704552] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8039.708676] Lustre: Skipped 1 previous similar message [ 8039.896140] Lustre: lustre-MDT0001: new disk, initializing [ 8040.168327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8040.199411] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8044.818266] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8056.326973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8056.638644] Lustre: lustre-OST0000: new disk, initializing [ 8056.641876] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8056.648225] Lustre: 267164:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8058.775888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8058.787282] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8058.894200] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8065.161898] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8075.591854] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8075.879770] Lustre: lustre-OST0001: new disk, initializing [ 8075.890434] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8075.897800] Lustre: 268035:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8077.831865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8077.838398] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8077.962623] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8083.894254] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8096.281294] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 15:05:07 (1786820707) [ 8111.312151] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 15:05:22 (1786820722) [ 8128.994288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8129.015682] Lustre: Skipped 5 previous similar messages [ 8129.023996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8129.030868] Lustre: Skipped 4 previous similar messages [ 8133.224877] Lustre: server umount lustre-OST0000 complete [ 8137.697989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8143.438157] Lustre: server umount lustre-MDT0000 complete [ 8147.995676] LustreError: 265568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820760 with bad export cookie 15333903815572803361 [ 8148.000686] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8148.003868] LustreError: 265568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8154.629599] Lustre: server umount lustre-MDT0001 complete [ 8165.704501] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 8238.302930] Lustre: server umount lustre-OST0001 complete [ 8246.756107] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 8262.958680] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 8303.409720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8303.761942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8303.796212] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8303.885860] Lustre: lustre-MDT0000: new disk, initializing [ 8303.986408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8303.990202] Lustre: Skipped 3 previous similar messages [ 8304.008333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8308.934743] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8319.875228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8319.985080] Lustre: 272835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8320.048234] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8320.053103] Lustre: Skipped 1 previous similar message [ 8320.184807] Lustre: lustre-MDT0001: new disk, initializing [ 8320.353665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8320.370025] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8325.158779] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8335.075354] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8340.317787] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8348.300908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8348.636244] Lustre: 274803:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8350.404391] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8350.706154] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8356.388405] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8368.219399] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8376.289159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8376.303740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8376.307433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8376.319282] Lustre: Skipped 6 previous similar messages [ 8376.337767] Lustre: Skipped 7 previous similar messages [ 8380.722904] Lustre: server umount lustre-OST0000 complete [ 8391.157532] Lustre: server umount lustre-MDT0000 complete [ 8391.655173] LustreError: 273690:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8391.665389] LustreError: 273690:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [ 8394.727120] LustreError: 272829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821006 with bad export cookie 15333903815572804831 [ 8394.728895] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8394.737036] LustreError: 272829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8395.140981] Lustre: server umount lustre-MDT0001 complete [ 8408.287447] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 8411.410889] Key type lgssc unregistered [ 8411.683335] LNet: 276566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8411.690451] LNetError: 276566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8411.706769] LNet: Removed LNI 192.168.201.104@tcp [ 8412.408742] Key type .llcrypt unregistered [ 8412.411146] Key type ._llcrypt unregistered [ 8439.075380] Key type ._llcrypt registered [ 8439.078279] Key type .llcrypt registered [ 8439.194177] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 8440.156252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8440.195900] alg: No test for adler32 (adler32-zlib) [ 8441.451823] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 8441.730700] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 8443.391555] Key type lgssc registered [ 8444.551898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8452.130158] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8452.147549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8453.614391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8457.736894] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8463.852627] LustreError: 278033:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8463.870580] LustreError: 278033:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8466.114141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8466.335273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8469.978386] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8476.545361] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8481.654149] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8487.715849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8488.085092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8494.022557] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8498.361338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8502.299541] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8503.351315] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8506.856841] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8506.867166] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8506.881241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8508.904337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8508.909037] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8508.910536] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8508.913498] Lustre: Skipped 1 previous similar message [ 8511.150653] Lustre: server umount lustre-OST0000 complete [ 8517.600692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8517.613453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8517.634362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8524.258575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8524.264209] Lustre: Skipped 3 previous similar messages [ 8529.378154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8529.387361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8529.404361] Lustre: Skipped 1 previous similar message [ 8529.518631] Lustre: server umount lustre-MDT0000 complete [ 8533.483281] LustreError: 278013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821145 with bad export cookie 8861226481176585569 [ 8533.490411] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8533.494361] LustreError: 278013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8533.917063] Lustre: server umount lustre-MDT0001 complete [ 8545.846744] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 8548.283390] Key type lgssc unregistered [ 8548.582914] LNet: 281621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8548.598298] LNetError: 281621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8548.615587] LNet: Removed LNI 192.168.201.104@tcp [ 8549.564070] Key type .llcrypt unregistered [ 8549.566202] Key type ._llcrypt unregistered [ 8567.003695] Key type ._llcrypt registered [ 8567.012356] Key type .llcrypt registered [ 8567.229469] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 8568.690223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8568.706502] alg: No test for adler32 (adler32-zlib) [ 8569.845956] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 8570.157290] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 8571.967346] Key type lgssc registered [ 8573.310399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8583.182449] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8583.214552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8598.815856] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8603.935481] LustreError: 282816:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.104@tcp: failed processing log, type 4: rc = -110 [ 8630.623336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8636.809527] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8643.828742] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8645.706250] Lustre: server umount lustre-OST0000 complete [ 8660.290153] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 8662.915247] Key type lgssc unregistered [ 8663.156627] LNet: 284451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8663.172692] LNetError: 284451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8663.190232] LNet: Removed LNI 192.168.201.104@tcp [ 8664.148258] Key type .llcrypt unregistered [ 8664.150524] Key type ._llcrypt unregistered [ 8706.160841] Key type ._llcrypt registered [ 8706.163206] Key type .llcrypt registered [ 8706.245221] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 8707.332871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8707.345900] alg: No test for adler32 (adler32-zlib) [ 8708.434797] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 8708.722569] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 8710.383178] Key type lgssc registered [ 8711.393709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8726.832899] loop0: detected capacity change from 0 to 204800000 [ 8727.098386] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8727.133959] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8727.612162] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8728.609396] loop0: detected capacity change from 0 to 204800000 [ 8728.982352] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8728.991294] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8729.606871] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8730.909540] loop0: detected capacity change from 0 to 204800000 [ 8731.244370] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8731.262643] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8731.370058] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8732.479093] loop0: detected capacity change from 0 to 204800000 [ 8732.775082] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8732.785829] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8733.048574] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8734.136322] loop0: detected capacity change from 0 to 204800000 [ 8734.176912] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8739.194883] loop0: detected capacity change from 0 to 204800000 [ 8739.260394] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8744.550205] loop0: detected capacity change from 0 to 204800000 [ 8744.569174] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8749.323829] loop0: detected capacity change from 0 to 204800000 [ 8749.360490] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8754.155312] loop0: detected capacity change from 0 to 204800000 [ 8754.310253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8754.470750] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8754.592452] Lustre: 287868:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8754.618750] Lustre: 287868:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8754.639477] Lustre: 287868:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8754.658449] Lustre: 287868:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 8754.672249] Lustre: 287868:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 8754.699036] Lustre: 287868:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8754.707258] Lustre: 287868:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 8754.719442] Lustre: 287868:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 8754.727761] Lustre: 287868:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 8754.736698] Lustre: 287868:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 8754.743574] Lustre: 287868:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 8754.751138] Lustre: 287868:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 8754.758520] Lustre: 287868:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 8754.766797] Lustre: 287868:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 8761.055777] Lustre: server umount MGS complete [ 8761.959437] loop0: detected capacity change from 0 to 204800000 [ 8762.055884] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8762.394137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8762.423173] Lustre: lustre-MDT0000: Aborting client recovery [ 8763.453592] loop1: detected capacity change from 0 to 204800000 [ 8763.581080] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8763.639753] Lustre: 288300:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8763.650960] Lustre: 288300:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8763.665319] Lustre: 288300:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 8763.680866] Lustre: 288300:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8763.688204] Lustre: 288300:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8763.693682] Lustre: 288300:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 8763.698909] Lustre: 288300:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 8763.709184] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8763.842574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8763.890858] Lustre: lustre-MDT0001: Aborting client recovery [ 8764.761318] loop2: detected capacity change from 0 to 204800000 [ 8765.016447] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8765.121559] Lustre: 288380:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8765.129458] Lustre: 288380:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8765.143284] Lustre: 288380:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8765.150271] Lustre: 288380:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8765.157968] Lustre: 288380:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8765.173794] Lustre: 288380:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8765.184933] Lustre: 288380:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8765.383807] Lustre: Found index 0 for lustre-OST0000, updating log [ 8765.458575] Lustre: 288399:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8765.570702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8765.602265] Lustre: lustre-OST0000: Aborting recovery [ 8766.791747] loop3: detected capacity change from 0 to 204800000 [ 8766.955742] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8767.029692] Lustre: 288464:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8767.045728] Lustre: 288464:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8767.068647] Lustre: 288464:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8767.086744] Lustre: 288464:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8767.107462] Lustre: 288464:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8767.114376] Lustre: 288464:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8767.121793] Lustre: 288464:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8767.136958] Lustre: Found index 1 for lustre-OST0001, updating log [ 8767.168700] Lustre: 288469:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8767.251872] Lustre: lustre-OST0001: Aborting recovery [ 8772.391808] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8772.397619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8772.411665] LustreError: 288244:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 8772.447280] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 8772.482572] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 8773.433893] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 8773.435383] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 8776.513515] LustreError: 288211:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 399us: rc = -2 [ 8777.022637] LustreError: 288212:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 544us: rc = -2 [ 8777.040979] LustreError: 288212:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 22 previous similar messages [ 8794.081617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8794.082269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8794.115037] Lustre: Skipped 2 previous similar messages [ 8794.125452] Lustre: Skipped 1 previous similar message [ 8799.199832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8799.205853] Lustre: Skipped 2 previous similar messages [ 8804.319140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8804.331664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8804.339232] Lustre: Skipped 6 previous similar messages [ 8804.511737] Lustre: server umount lustre-MDT0000 complete [ 8805.667784] LustreError: 288194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821417 with bad export cookie 13505132057111715071 [ 8805.698823] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8809.441944] LustreError: 288477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8809.445301] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8809.449287] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8809.464900] LustreError: 288477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8809.498545] Lustre: Skipped 2 previous similar messages [ 8810.469542] LustreError: 288749:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8810.491270] LustreError: 288749:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8812.142512] Lustre: server umount lustre-MDT0001 complete [ 8827.359289] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8827.536038] Lustre: server umount lustre-OST0000 complete [ 8834.776613] Lustre: server umount lustre-OST0001 complete [ 8842.586492] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 15:17:34 (1786821454) [ 8872.401670] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 8883.884718] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 8919.094752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8919.415574] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8919.452611] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8919.539469] Lustre: lustre-MDT0000: new disk, initializing [ 8919.598271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8919.603427] Lustre: Skipped 1 previous similar message [ 8919.614780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8924.054444] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8934.375764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8934.486661] Lustre: 292538:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8934.555486] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8934.564436] Lustre: Skipped 1 previous similar message [ 8934.636386] Lustre: lustre-MDT0001: new disk, initializing [ 8934.821311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8934.871201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8934.895964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8939.774241] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8948.762980] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8954.449375] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8960.924562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8961.208830] Lustre: lustre-OST0000: new disk, initializing [ 8961.212731] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8961.218762] Lustre: 294509:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8961.298061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8966.219268] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 8969.803944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8969.822398] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8969.941273] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8975.684918] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8981.208350] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8994.751238] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9002.511564] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9012.500808] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9062.367957] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9062.370934] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9062.382304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9062.388395] Lustre: Skipped 1 previous similar message [ 9062.408844] Lustre: Skipped 4 previous similar messages [ 9064.311077] Lustre: server umount lustre-OST0000 complete [ 9072.612165] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9072.614374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9072.627232] Lustre: Skipped 1 previous similar message [ 9072.641696] Lustre: Skipped 1 previous similar message [ 9074.591200] Lustre: server umount lustre-MDT0000 complete [ 9077.732645] LustreError: 293764:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9077.753737] LustreError: 293764:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9078.558730] LustreError: 292530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821690 with bad export cookie 13505132057111717780 [ 9078.565478] LustreError: 292530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9078.574046] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9078.937288] Lustre: server umount lustre-MDT0001 complete [ 9087.350196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9092.600716] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9097.253219] Lustre: server umount MGS complete [ 9108.424750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9109.037625] LustreError: 298495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9109.067249] LustreError: 298508:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9109.132483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9113.211576] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9114.594898] LustreError: 298496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9121.855950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9127.427845] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9134.346378] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9140.012798] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9146.564199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9146.903891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9146.910891] Lustre: Skipped 1 previous similar message [ 9153.373111] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9160.642360] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9188.832206] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9188.849842] LustreError: Skipped 1 previous similar message [ 9188.862502] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9188.880909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9202.143740] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9202.284832] Lustre: server umount lustre-OST0000 complete [ 9209.315262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9209.328674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9209.350383] Lustre: Skipped 1 previous similar message [ 9212.989129] Lustre: server umount lustre-MDT0000 complete [ 9215.462717] LustreError: 301399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9215.498917] LustreError: 301399:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9217.346327] LustreError: 298476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821829 with bad export cookie 13505132057111719719 [ 9217.350848] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9217.355706] LustreError: 298476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9217.666481] Lustre: server umount lustre-MDT0001 complete [ 9229.772778] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 9232.927263] Key type lgssc unregistered [ 9233.343705] LNet: 302445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9233.347978] LNetError: 302445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9233.388415] LNet: Removed LNI 192.168.201.104@tcp [ 9234.279806] Key type .llcrypt unregistered [ 9234.285933] Key type ._llcrypt unregistered [ 9276.375623] Key type ._llcrypt registered [ 9276.377349] Key type .llcrypt registered [ 9276.492745] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 9292.143643] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 9293.188900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9293.345141] alg: No test for adler32 (adler32-zlib) [ 9294.517248] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 9294.926409] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 9296.607117] Key type lgssc registered [ 9297.776312] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9331.973338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9331.988131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9333.172695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9333.196979] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9333.258200] Lustre: lustre-MDT0000: new disk, initializing [ 9333.329738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9333.376153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9337.867121] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9347.878937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9348.037898] Lustre: 306410:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9348.093858] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9348.099232] Lustre: Skipped 1 previous similar message [ 9348.160522] Lustre: lustre-MDT0001: new disk, initializing [ 9348.265911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9348.306471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9348.323773] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9353.179149] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9361.928555] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9367.137626] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9373.809874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9374.268311] Lustre: lustre-OST0000: new disk, initializing [ 9374.275615] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9374.283476] Lustre: 308380:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9374.391806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9382.546119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9382.585910] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9382.795103] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9383.213366] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9394.140192] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9401.711805] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9413.352891] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9420.922086] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9426.604287] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9464.802250] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9464.809844] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9464.822576] LustreError: Skipped 1 previous similar message [ 9464.828023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9464.844609] Lustre: Skipped 1 previous similar message [ 9469.630794] Lustre: server umount lustre-OST0000 complete [ 9475.045944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9475.062308] Lustre: Skipped 1 previous similar message [ 9475.068431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9475.072964] Lustre: Skipped 1 previous similar message [ 9480.160175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9480.166666] Lustre: Skipped 1 previous similar message [ 9485.284054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9485.296314] Lustre: Skipped 1 previous similar message [ 9488.351113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9488.575609] Lustre: server umount lustre-MDT0000 complete [ 9492.799928] LustreError: 306403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822104 with bad export cookie 6379668331733876079 [ 9492.804262] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9492.816779] LustreError: 306403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9493.340877] Lustre: server umount lustre-MDT0001 complete [ 9502.213933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9507.522677] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9511.811404] Lustre: server umount MGS complete [ 9521.982240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9522.545908] LustreError: 312292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9522.559518] LustreError: 312292:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9522.576765] LustreError: 312305:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9522.637660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9527.176661] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9527.778750] LustreError: 312293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9531.880862] LustreError: 312292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9534.847415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9535.112183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9538.871177] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9546.173767] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9551.043989] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9557.708684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9558.048946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9565.043534] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9573.022953] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9600.994640] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9601.003964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9601.019741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9603.042886] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9603.049127] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9606.835286] Lustre: server umount lustre-OST0000 complete [ 9611.753697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9611.761889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9611.773582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9611.775902] Lustre: Skipped 2 previous similar messages [ 9616.728685] Lustre: server umount lustre-MDT0000 complete [ 9618.405604] LustreError: 312292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9618.429336] LustreError: 312292:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9620.694981] LustreError: 312274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822232 with bad export cookie 6379668331733877906 [ 9620.702166] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9620.714921] LustreError: 312274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9621.238413] Lustre: server umount lustre-MDT0001 complete [ 9637.433902] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [ 9641.106617] Key type lgssc unregistered [ 9641.586140] LNet: 316243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9641.602360] LNetError: 316243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9641.663223] LNet: Removed LNI 192.168.201.104@tcp [ 9642.829721] Key type .llcrypt unregistered [ 9642.831849] Key type ._llcrypt unregistered [ 9664.413275] Key type ._llcrypt registered [ 9664.415526] Key type .llcrypt registered [ 9664.577056] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [ 9680.608368] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 9681.832507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9682.111140] alg: No test for adler32 (adler32-zlib) [ 9683.243905] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 9683.511918] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [ 9685.231907] Key type lgssc registered [ 9686.816270] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9711.713947] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [ 9726.173534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9726.203964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9727.470772] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9727.557967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9727.699697] Lustre: lustre-MDT0000: new disk, initializing [ 9727.782185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9727.804762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9732.528588] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [ 9743.380959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9743.666494] Lustre: lustre-OST0000: new disk, initializing [ 9743.672839] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9743.678051] Lustre: Skipped 1 previous similar message [ 9743.686757] Lustre: 320296:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9743.788686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9744.078469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9744.091732] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9744.142715] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9750.447791] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10571.302786] Lustre: server umount lustre-MDT0000 complete [10575.112949] LustreError: 319482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823187 with bad export cookie 4683532375751941321 [10575.122485] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10585.802885] Lustre: server umount lustre-OST0000 complete [10612.815092] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [10625.199861] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [10664.200804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10664.622286] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10664.674519] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10664.787559] Lustre: lustre-MDT0000: new disk, initializing [10664.941899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10664.986967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10670.030676] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10681.950231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10682.115943] Lustre: 324633:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10682.161197] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10682.164878] Lustre: Skipped 1 previous similar message [10682.270271] Lustre: lustre-MDT0001: new disk, initializing [10682.408053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10682.473449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10682.488049] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10688.791795] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10699.658089] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10707.024891] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10715.022925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10715.550502] Lustre: lustre-OST0000: new disk, initializing [10715.566772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10715.582989] Lustre: 326603:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10715.757301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10722.460358] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10724.443783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10724.456347] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10724.622661] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10732.549417] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10739.084596] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10739.235827] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10744.110494] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10744.362835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10750.432830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10750.434941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10750.454412] LustreError: Skipped 1 previous similar message [10750.455879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10750.483048] Lustre: Skipped 1 previous similar message [10752.895830] Lustre: server umount lustre-OST0000 complete [10759.144882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10759.161974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10759.190796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10759.202798] Lustre: Skipped 1 previous similar message [10760.674823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10760.678808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10760.706692] Lustre: Skipped 1 previous similar message [10763.345603] Lustre: server umount lustre-MDT0000 complete [10767.167870] LustreError: 324627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823379 with bad export cookie 4683532375754882133 [10767.174628] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10767.178721] LustreError: 324627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10767.637981] Lustre: server umount lustre-MDT0001 complete [10771.416677] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [10773.363830] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 15:49:44 (1786823384) [10781.317632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10781.897765] LustreError: 328914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10781.912765] LustreError: 328914:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10782.008248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10787.299480] LustreError: 328915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10787.351543] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10792.424546] LustreError: 328914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10796.077222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10796.586709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10802.152981] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10809.734549] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10815.458050] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10822.811732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10823.242659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10829.705712] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10836.724845] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10842.970298] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10843.103360] Lustre: lustre-OST0001: new disk, initializing [10843.106535] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10843.110921] Lustre: 331750:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10843.178138] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10849.885404] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10852.964624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10852.971927] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10853.035986] LustreError: 332230:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10854.123666] LustreError: 332230:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10854.134612] LustreError: 332230:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [10855.150109] LustreError: 332236:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10857.267196] LustreError: 332236:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10857.280469] LustreError: 332236:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 4 previous similar messages [10859.269109] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10859.556555] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10861.429485] LustreError: 332230:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10861.452470] LustreError: 332230:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 6 previous similar messages [10864.912222] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8d3d10c95800.ost_server_uuid 50 [10866.556349] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8d3d10c95800.ost_server_uuid in FULL state after 0 sec [10869.674304] LustreError: 332236:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10869.685889] LustreError: 332236:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [10871.715352] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d3d10c95800.ost_server_uuid 50 [10873.339851] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d3d10c95800.ost_server_uuid in FULL state after 0 sec [10874.869616] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10889.198291] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10889.207606] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10889.221798] Lustre: Skipped 1 previous similar message [10889.228607] Lustre: Skipped 1 previous similar message [10894.312225] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10894.328861] Lustre: Skipped 2 previous similar messages [10895.156544] Lustre: server umount lustre-OST0001 complete [10895.474150] LustreError: 330737:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10903.015805] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10903.028834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10903.055554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10904.566672] LustreError: 330737:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10904.604297] LustreError: 330737:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10907.576487] Lustre: server umount lustre-OST0000 complete [10914.275404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10914.282620] LustreError: Skipped 1 previous similar message [10914.291322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10914.308549] Lustre: Skipped 1 previous similar message [10918.298445] Lustre: server umount lustre-MDT0000 complete [10919.910858] LustreError: 328914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10919.932903] LustreError: 328914:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10922.684826] LustreError: 328896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823534 with bad export cookie 4683532375754883022 [10922.697801] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10923.081881] Lustre: server umount lustre-MDT0001 complete [10934.073353] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [10937.224851] Key type lgssc unregistered [10937.648055] LNet: 334445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10937.651910] LNetError: 334445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10937.666418] LNet: Removed LNI 192.168.201.104@tcp [10938.473565] Key type .llcrypt unregistered [10938.476820] Key type ._llcrypt unregistered [10967.491462] Key type ._llcrypt registered [10967.493641] Key type .llcrypt registered [10967.630707] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [10969.018503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10969.429221] alg: No test for adler32 (adler32-zlib) [10970.760922] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [10971.073560] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [10972.759250] Key type lgssc registered [10974.552154] Lustre: Echo OBD driver; http://www.lustre.org/ [10983.499437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10983.512858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10984.931707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10988.866531] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [10994.146097] LustreError: 335867:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10994.163614] LustreError: 335867:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10996.156920] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10999.281421] LustreError: 335868:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11003.735624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11004.226098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11009.260687] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11020.016455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11020.385709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11028.160443] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11029.612415] LustreError: 337548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11029.640542] LustreError: 337548:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11029.656898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [11035.120923] LustreError: 337981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11035.153557] LustreError: 337981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11035.775805] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11040.231935] LustreError: 337981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11040.250178] LustreError: 337981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11045.256966] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11045.581701] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11051.004890] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [11053.032534] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11061.410772] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11070.118219] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [11070.386191] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11184.097202] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11184.100506] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11184.112295] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11184.123744] LustreError: Skipped 1 previous similar message [11184.147855] Lustre: Skipped 1 previous similar message [11186.416594] Lustre: server umount lustre-OST0001 complete [11189.231235] LustreError: 337549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11189.257846] LustreError: 337549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11193.827179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11193.828057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11193.840842] LustreError: Skipped 1 previous similar message [11193.842188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11193.857384] Lustre: Skipped 1 previous similar message [11193.880732] Lustre: Skipped 2 previous similar messages [11199.460329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11199.478440] Lustre: Skipped 3 previous similar messages [11204.576884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11207.135192] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11207.290723] Lustre: server umount lustre-OST0000 complete [11214.304280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11214.310312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11214.322733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11214.325213] Lustre: Skipped 1 previous similar message [11216.875378] Lustre: server umount lustre-MDT0000 complete [11219.941889] LustreError: 335868:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11219.966058] LustreError: 335868:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [11219.988470] LustreError: 339550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823832 with bad export cookie 4958290799441158237 [11219.993934] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11219.999287] LustreError: 339550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11220.379409] Lustre: server umount lustre-MDT0001 complete [11231.572391] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [11234.335841] Key type lgssc unregistered [11234.822740] LNet: 340813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11234.841377] LNetError: 340813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11234.882563] LNet: Removed LNI 192.168.201.104@tcp [11236.018404] Key type .llcrypt unregistered [11236.020299] Key type ._llcrypt unregistered [11290.816666] Key type ._llcrypt registered [11290.818125] Key type .llcrypt registered [11290.956457] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [11292.404399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11292.746801] alg: No test for adler32 (adler32-zlib) [11293.877664] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [11294.079946] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [11295.767503] Key type lgssc registered [11297.540886] Lustre: Echo OBD driver; http://www.lustre.org/ [11307.049422] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11307.063067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11308.395636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11313.003684] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11318.761596] LustreError: 343108:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11318.793559] LustreError: 343108:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11321.533187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11321.890688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11326.532263] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11329.960258] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11337.753818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11338.157142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11345.349567] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11346.348293] LustreError: 344595:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11351.528825] LustreError: 345017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11353.579638] LustreError: 345017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11353.601865] LustreError: 345017:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11353.618255] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [11355.057533] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11355.426993] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11360.774634] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [11362.258124] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11371.166976] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11377.301688] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11391.463230] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11391.469454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11391.485981] Lustre: Skipped 3 previous similar messages [11391.500694] Lustre: Skipped 3 previous similar messages [11394.805073] Lustre: server umount lustre-MDT0000 complete [11396.577159] LustreError: 343107:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11398.698174] LustreError: 346118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824010 with bad export cookie 3963779729293221560 [11398.705022] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11398.708928] LustreError: 346118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11399.078416] Lustre: server umount lustre-MDT0001 complete [11412.132918] Lustre: server umount lustre-OST0000 complete [11425.425678] Lustre: server umount lustre-OST0001 complete [11444.473729] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [11455.887200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11456.356517] LustreError: 348385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11456.372802] LustreError: 348385:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11456.436085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11461.181503] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11469.302514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11470.066001] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11474.947651] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11477.986602] Lustre: 349525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11484.688587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11491.451950] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11495.408204] LustreError: 349882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11495.433656] LustreError: 349882:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11500.055687] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11500.455872] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11500.466735] Lustre: Skipped 1 previous similar message [11501.509547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [11501.529826] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [11506.557178] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11514.710869] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11519.846224] Lustre: 351400:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11526.112576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11526.119639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11526.135074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11527.138677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11527.143619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11527.175100] Lustre: Skipped 1 previous similar message [11530.703605] Lustre: server umount lustre-MDT0000 complete [11532.275835] LustreError: 349398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11532.298912] LustreError: 349398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11534.784595] LustreError: 348367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824146 with bad export cookie 3963779729293223261 [11534.805932] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11534.808198] LustreError: 348367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11535.486133] Lustre: server umount lustre-MDT0001 complete [11550.299986] Lustre: server umount lustre-OST0000 complete [11553.184691] Lustre: 342373:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786824149/real 1786824149] req@ffff9b6d5f393800 x1873620664174208/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786824165 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11553.218360] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11553.228214] Lustre: Skipped 1 previous similar message [11554.403893] Lustre: server umount lustre-OST0001 complete [11557.941688] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [11559.532115] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [11561.183621] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 16:02:52 (1786824172) [11584.328138] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [11595.751458] loop0: detected capacity change from 0 to 1099511627776 [11596.679669] SGI XFS with ACLs, security attributes, quota, no debug enabled [11596.687193] *** VALIDATE xfs *** [11596.705316] XFS (loop0): Mounting V5 Filesystem [11596.754726] XFS (loop0): Ending clean mount [11598.299792] loop1: detected capacity change from 0 to 18691697672192 [11708.099486] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [11724.180870] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [11761.729642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11762.017394] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11762.080188] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11762.175194] Lustre: lustre-MDT0000: new disk, initializing [11762.247381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11762.272662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11766.470982] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11777.360688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11777.544379] Lustre: 356559:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11777.618940] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11777.623164] Lustre: Skipped 1 previous similar message [11777.705750] Lustre: lustre-MDT0001: new disk, initializing [11777.838640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11777.861956] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11783.229913] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11795.858746] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11801.885090] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11809.689269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11810.026320] Lustre: lustre-OST0000: new disk, initializing [11810.029694] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11810.037608] Lustre: 358527:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11811.510130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11811.548802] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11811.674392] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11818.810579] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11828.527921] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11833.005733] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11833.236870] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11837.532258] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11837.927817] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11842.543965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11842.545678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11842.562410] Lustre: Skipped 1 previous similar message [11842.563457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11842.563463] Lustre: Skipped 3 previous similar messages [11842.592315] LustreError: Skipped 1 previous similar message [11847.657636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11847.661864] Lustre: Skipped 2 previous similar messages [11854.304145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11854.525345] Lustre: server umount lustre-OST0000 complete [11859.941779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11859.963768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11859.972039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11859.980000] Lustre: Skipped 2 previous similar messages [11865.073627] Lustre: server umount lustre-MDT0000 complete [11868.130550] LustreError: 356566:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11868.148564] LustreError: 356566:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11868.487079] LustreError: 358526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824480 with bad export cookie 3963779729293224815 [11868.491495] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11868.507600] LustreError: 358526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11869.052286] Lustre: server umount lustre-MDT0001 complete [11876.605451] XFS (loop0): Unmounting Filesystem [11881.624285] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 16:08:13 (1786824493) [11887.913981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11888.477565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11888.491483] Lustre: Skipped 2 previous similar messages [11892.526665] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11901.436206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11906.951352] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11914.069776] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11918.758505] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11927.314631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11936.282928] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [11946.142830] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11956.706451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11956.727485] Lustre: Skipped 1 previous similar message [11956.739207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11956.750074] Lustre: Skipped 3 previous similar messages [11961.347397] Lustre: server umount lustre-OST0000 complete [11971.556126] Lustre: server umount lustre-MDT0000 complete [11975.027779] LustreError: 361070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824587 with bad export cookie 3963779729293225690 [11975.034593] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11975.244297] Lustre: server umount lustre-MDT0001 complete [11988.357153] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [11991.875280] Key type lgssc unregistered [11992.355971] LNet: 364719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11992.372867] LNetError: 364719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11992.412532] LNet: Removed LNI 192.168.201.104@tcp [11993.760132] Key type .llcrypt unregistered [11993.761726] Key type ._llcrypt unregistered [12025.696606] Key type ._llcrypt registered [12025.699402] Key type .llcrypt registered [12025.811294] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [12027.030262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12027.216348] alg: No test for adler32 (adler32-zlib) [12028.249741] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [12028.531499] LNet: Added LNI 192.168.201.104@tcp [8/256/0/180] [12030.287257] Key type lgssc registered [12031.612958] Lustre: Echo OBD driver; http://www.lustre.org/ [12039.146135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12039.180133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12040.862729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12045.336753] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12050.404649] LustreError: 366289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12050.422771] LustreError: 366289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12053.998522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12054.370672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12059.011331] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12067.479653] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12073.521933] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12080.266519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12080.557395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12086.869833] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12094.019723] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12105.700096] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12105.705779] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12105.727062] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12106.849657] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.4@tcp (stopping) [12109.281652] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12109.286161] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12109.312422] Lustre: Skipped 1 previous similar message [12111.210370] Lustre: server umount lustre-MDT0001 complete [12111.981084] LustreError: 366284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12114.402629] LustreError: 366289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12114.429274] LustreError: 366289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12117.083670] LustreError: 366284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12121.747450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12121.878687] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12121.883761] Lustre: Found index 1 for lustre-MDT0001, updating log [12121.911429] Lustre: 366276:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12122.251422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12126.841734] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12127.722099] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12127.743364] LustreError: 369769:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12127.753928] LustreError: 369769:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12127.778106] LustreError: 366280:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12127.917154] LustreError: 369784:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.104@tcp: cfg command failed: rc = -17 [12127.934053] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [12127.934053] [12142.711863] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.4@tcp (stopping) [12143.073319] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12144.384478] Lustre: server umount lustre-MDT0001 complete [12147.789694] LustreError: 366285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12147.827772] LustreError: 366285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12155.612195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12155.687894] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12155.692289] Lustre: Found index 1 for lustre-MDT0001, updating log [12155.749380] Lustre: 366277:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12156.071227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12161.032437] LustreError: 370784:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12161.048120] LustreError: 370784:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12161.057117] LustreError: 366280:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12161.072465] LustreError: 366280:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12161.249232] LustreError: 370797:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [12161.265181] LustreError: 370797:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [12161.275561] LustreError: 370797:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.104@tcp: cfg command failed: rc = -17 [12161.282383] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.201.104@tcp [12161.282383] [12161.415889] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12161.532740] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12468.708535] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12468.710887] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12468.722979] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12468.722987] Lustre: Skipped 2 previous similar messages [12468.748236] Lustre: Skipped 2 previous similar messages [12475.061763] Lustre: server umount lustre-MDT0001 complete [12475.480073] LustreError: 366283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12475.511235] LustreError: 366283:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12486.377482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12486.427121] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12486.428795] Lustre: Found index 1 for lustre-MDT0001, updating log [12486.451872] Lustre: 366276:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12486.658741] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12489.995231] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12491.776682] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12495.876697] LustreError: 379698:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12495.893425] LustreError: 379698:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12495.922552] LustreError: 366280:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12495.932209] LustreError: 366280:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12496.077700] LustreError: 379701:0:(lod_lov.c:210:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [12496.112431] LustreError: 379701:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.104@tcp: cfg command failed: rc = -114 [12496.125386] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [12496.125386] [12814.304572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12814.308762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12814.315703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12814.315713] Lustre: Skipped 3 previous similar messages [12814.342894] Lustre: Skipped 1 previous similar message [12825.055175] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12825.322193] Lustre: server umount lustre-MDT0000 complete [12829.223950] LustreError: 366270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825441 with bad export cookie 10400647280076627894 [12829.228034] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12829.240585] LustreError: 366270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12829.665436] LustreError: 366283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12829.666830] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12829.692510] LustreError: 366283:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [12829.722294] Lustre: Skipped 1 previous similar message [12834.785192] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12834.793667] Lustre: Skipped 9 previous similar messages [12835.623718] Lustre: server umount lustre-MDT0001 complete [12839.326105] Lustre: server umount lustre-OST0000 complete [12847.745166] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [12857.263647] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [12894.086659] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 16:25:05 (1786825505) [12900.310436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12900.598849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12900.647766] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12900.755839] Lustre: lustre-MDT0000: new disk, initializing [12900.890971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12900.914465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12905.383626] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12914.881681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12914.962801] Lustre: 390318:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12915.064142] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12915.067174] Lustre: Skipped 1 previous similar message [12915.122642] Lustre: lustre-MDT0001: new disk, initializing [12915.203316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12915.248387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12915.268866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12919.281288] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12930.325438] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12936.334178] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12943.651422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12944.080505] Lustre: lustre-OST0000: new disk, initializing [12944.083123] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12944.088046] Lustre: 392289:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12944.166672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12950.177925] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_default_debug -1 all [12951.154058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12951.169939] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12951.349211] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12959.960674] Lustre: DEBUG MARKER: oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12966.371720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12966.379555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12966.395027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12972.146387] Lustre: server umount lustre-MDT0000 complete [12975.630130] LustreError: 390310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825587 with bad export cookie 10400647280076629847 [12975.632201] LustreError: MGC192.168.201.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12975.634940] LustreError: 391126:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [12975.653483] LustreError: 390310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12975.883330] Lustre: server umount lustre-MDT0001 complete [12989.815809] Lustre: server umount lustre-OST0000 complete [13017.768993] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing set_hostid [13033.583753] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing load_modules_local [13062.880719] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12802 sec ========= 16:27:54 (1786825674) [13064.588972] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:27:55 (1786825675) === [13068.304711] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:27:59 (1786825679) === [13108.621453] Lustre: DEBUG MARKER: oleg104-server.virtnet: executing unload_modules_local [13111.390568] Key type lgssc unregistered [13111.691789] LNet: 398677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13111.701900] LNetError: 398677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13111.724458] LNet: Removed LNI 192.168.201.104@tcp [13112.416784] Key type .llcrypt unregistered [13112.418413] Key type ._llcrypt unregistered