[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 440782936 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003275] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005008] kvm-guest: setup PV IPIs [ 0.007443] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008014] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009006] pid_max: default: 32768 minimum: 301 [ 0.011111] LSM: Security Framework initializing [ 0.012033] Yama: becoming mindful. [ 0.013023] SELinux: Initializing. [ 0.014048] *** VALIDATE selinux *** [ 0.021552] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025696] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026138] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028128] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029174] *** VALIDATE tmpfs *** [ 0.031640] *** VALIDATE proc *** [ 0.032390] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.034344] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035208] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037033] Spectre V2 : User space: Vulnerable [ 0.038006] Speculative Store Bypass: Vulnerable [ 0.041654] debug: unmapping init [mem 0xffffffff88659000-0xffffffff88660fff] [ 0.043218] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044614] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045014] ... version: 2 [ 0.046009] ... bit width: 48 [ 0.047007] ... generic registers: 4 [ 0.048008] ... value mask: 0000ffffffffffff [ 0.049006] ... max period: 00007fffffffffff [ 0.050005] ... fixed-purpose events: 3 [ 0.050844] ... event mask: 000000070000000f [ 0.052233] rcu: Hierarchical SRCU implementation. [ 0.054343] smp: Bringing up secondary CPUs ... [ 0.055406] x86: Booting SMP configuration: [ 0.056011] .... node #0, CPUs: #1 #2 #3 [ 0.059296] smp: Brought up 1 node, 4 CPUs [ 0.060999] smpboot: Max logical packages: 1 [ 0.061986] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.274019] node 0 deferred pages initialised in 211ms [ 0.277323] devtmpfs: initialized [ 0.278277] x86/mm: Memory block size: 128MB [ 0.280954] gcov: version magic: 0x41383552 [ 0.282363] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.283106] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.285041] pinctrl core: initialized pinctrl subsystem [ 0.286160] [ 0.286488] ************************************************************* [ 0.288010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.290009] ** ** [ 0.292011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.294012] ** ** [ 0.297011] ** This means that this kernel is built to expose internal ** [ 0.299008] ** IOMMU data structures, which may compromise security on ** [ 0.301010] ** your system. ** [ 0.304010] ** ** [ 0.306008] ** If you see this message and you are not debugging the ** [ 0.309009] ** kernel, report this immediately to your vendor! ** [ 0.311009] ** ** [ 0.313020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.316009] ************************************************************* [ 0.318591] NET: Registered protocol family 16 [ 0.320467] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.322030] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.323025] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.326231] cpuidle: using governor menu [ 0.327355] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.328365] PCI: Using configuration type 1 for base access [ 0.330100] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.338098] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.339015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.342026] cryptd: max_cpu_qlen set to 1000 [ 0.344157] ACPI: Added _OSI(Module Device) [ 0.345007] ACPI: Added _OSI(Processor Device) [ 0.345834] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.346009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.348741] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.354366] ACPI: Interpreter enabled [ 0.355035] ACPI: PM: (supports S0 S3 S4 S5) [ 0.355858] ACPI: Using IOAPIC for interrupt routing [ 0.357057] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.358281] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.365000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.368020] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.369008] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.371051] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.374921] acpiphp: Slot [2] registered [ 0.375098] acpiphp: Slot [3] registered [ 0.377048] acpiphp: Slot [4] registered [ 0.378020] acpiphp: Slot [5] registered [ 0.379117] acpiphp: Slot [6] registered [ 0.381145] acpiphp: Slot [7] registered [ 0.381875] acpiphp: Slot [8] registered [ 0.383109] acpiphp: Slot [9] registered [ 0.384132] acpiphp: Slot [10] registered [ 0.385082] acpiphp: Slot [11] registered [ 0.385865] acpiphp: Slot [12] registered [ 0.387044] acpiphp: Slot [13] registered [ 0.387999] acpiphp: Slot [14] registered [ 0.389097] acpiphp: Slot [15] registered [ 0.391131] acpiphp: Slot [16] registered [ 0.392065] acpiphp: Slot [17] registered [ 0.392787] acpiphp: Slot [18] registered [ 0.393043] acpiphp: Slot [19] registered [ 0.393773] acpiphp: Slot [20] registered [ 0.394038] acpiphp: Slot [21] registered [ 0.394884] acpiphp: Slot [22] registered [ 0.396048] acpiphp: Slot [23] registered [ 0.397056] acpiphp: Slot [24] registered [ 0.397895] acpiphp: Slot [25] registered [ 0.399039] acpiphp: Slot [26] registered [ 0.399922] acpiphp: Slot [27] registered [ 0.400040] acpiphp: Slot [28] registered [ 0.400947] acpiphp: Slot [29] registered [ 0.403076] acpiphp: Slot [30] registered [ 0.404058] acpiphp: Slot [31] registered [ 0.405081] PCI host bridge to bus 0000:00 [ 0.406014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.408013] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.410020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.412012] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.415028] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.417027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.419174] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.423613] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.427712] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.438014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.442656] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.446017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.448014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.451017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.453481] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.456072] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.458023] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.459549] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.464018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.475015] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.478006] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.481131] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.487019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.491023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.502011] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.510568] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.517022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.526013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.540032] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.546894] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.551013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.557017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.569020] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.576895] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.581016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.589024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.606020] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.615397] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.619012] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.625020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.641024] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.649014] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.655023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.663016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.677035] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.688198] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.691358] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.692185] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.694305] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.695104] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.699029] iommu: Default domain type: Passthrough [ 0.700319] SCSI subsystem initialized [ 0.700959] ACPI: bus type USB registered [ 0.702089] usbcore: registered new interface driver usbfs [ 0.704045] usbcore: registered new interface driver hub [ 0.705081] usbcore: registered new device driver usb [ 0.707185] pps_core: LinuxPPS API ver. 1 registered [ 0.708007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.710060] PTP clock support registered [ 0.712040] EDAC MC: Ver: 3.0.0 [ 0.713412] PCI: Using ACPI for IRQ routing [ 0.715281] NetLabel: Initializing [ 0.717007] NetLabel: domain hash size = 128 [ 0.717933] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.719044] NetLabel: unlabeled traffic allowed by default [ 0.721069] vgaarb: loaded [ 0.722268] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.723009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.729339] clocksource: Switched to clocksource kvm-clock [ 0.885797] VFS: Disk quotas dquot_6.6.0 [ 0.887718] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.891421] *** VALIDATE ramfs *** [ 0.893010] *** VALIDATE hugetlbfs *** [ 0.894908] pnp: PnP ACPI init [ 0.897687] pnp: PnP ACPI: found 6 devices [ 0.913053] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.914726] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.915850] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.916948] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.918085] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.919596] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.921203] NET: Registered protocol family 2 [ 0.923149] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.926957] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.930047] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.934811] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.938598] TCP: Hash tables configured (established 65536 bind 65536) [ 0.941949] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.944975] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.946836] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.948831] NET: Registered protocol family 1 [ 0.951351] RPC: Registered named UNIX socket transport module. [ 0.952827] RPC: Registered udp transport module. [ 0.954330] RPC: Registered tcp transport module. [ 0.955285] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.956569] NET: Registered protocol family 44 [ 0.957391] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.959089] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.960361] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.962080] PCI: CLS 0 bytes, default 64 [ 0.963531] Unpacking initramfs... [ 2.465189] debug: unmapping init [mem 0xffff9c04fcc54000-0xffff9c04fffbffff] [ 2.471931] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.474455] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.479418] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.992321] Initialise system trusted keyrings [ 2.993855] Key type blacklist registered [ 2.996814] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.005649] zbud: loaded [ 3.008982] *** VALIDATE nfs *** [ 3.010061] *** VALIDATE nfs4 *** [ 3.011700] pstore: using deflate compression [ 3.018236] Platform Keyring initialized [ 3.189401] NET: Registered protocol family 38 [ 3.190805] Key type asymmetric registered [ 3.194290] Asymmetric key parser 'x509' registered [ 3.196322] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.199962] io scheduler mq-deadline registered [ 3.201630] io scheduler kyber registered [ 3.203103] io scheduler bfq registered [ 3.208154] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.210918] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.213385] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.216069] ACPI: Power Button [PWRF] [ 3.331816] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.434578] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.648503] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.747427] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.970367] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.007722] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.055438] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.063793] Non-volatile memory driver v1.3 [ 4.065158] Linux agpgart interface v0.103 [ 4.101248] virtio_blk virtio1: [vda] 132696 512-byte logical blocks (67.9 MB/64.8 MiB) [ 4.104163] vda: detected capacity change from 0 to 67940352 [ 4.122429] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.124919] vdb: detected capacity change from 0 to 1073741824 [ 4.155191] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.158339] vdc: detected capacity change from 0 to 2621440000 [ 4.176701] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.182616] vdd: detected capacity change from 0 to 2621440000 [ 4.217187] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.219727] vde: detected capacity change from 0 to 4294967296 [ 4.244451] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.246845] vdf: detected capacity change from 0 to 4294967296 [ 4.260351] libphy: Fixed MDIO Bus: probed [ 4.290636] usbcore: registered new interface driver usbserial_generic [ 4.294971] usbserial: USB Serial support registered for generic [ 4.298916] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.307817] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.311961] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.315376] mousedev: PS/2 mouse device common for all mice [ 4.319348] rtc_cmos 00:05: RTC can wake from S4 [ 4.323255] rtc_cmos 00:05: registered as rtc0 [ 4.325522] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.329289] intel_pstate: CPU model not supported [ 4.332458] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.338745] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.342391] hid: raw HID events driver (C) Jiri Kosina [ 4.346849] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.353057] usbcore: registered new interface driver usbhid [ 4.355308] usbhid: USB HID core driver [ 4.356955] drop_monitor: Initializing network drop monitor service [ 4.359369] Initializing XFRM netlink socket [ 4.361927] NET: Registered protocol family 10 [ 4.366064] Segment Routing with IPv6 [ 4.367468] NET: Registered protocol family 17 [ 4.370180] mpls_gso: MPLS GSO support [ 4.374958] RAS: Correctable Errors collector initialized. [ 4.376925] AVX version of gcm_enc/dec engaged. [ 4.378517] AES CTR mode by8 optimization enabled [ 4.492160] sched_clock: Marking stable (4492143509, 0)->(5176165081, -684021572) [ 4.496921] registered taskstats version 1 [ 4.501952] Loading compiled-in X.509 certificates [ 4.506996] zswap: loaded using pool lzo/zbud [ 4.532432] Key type big_key registered [ 4.552108] Key type encrypted registered [ 4.553534] ima: No TPM chip found, activating TPM-bypass! [ 4.556139] ima: Allocated hash algorithm: sha1 [ 4.559219] ima: No architecture policies found [ 4.562277] evm: Initialising EVM extended attributes: [ 4.563947] evm: security.selinux [ 4.565197] evm: security.ima [ 4.566292] evm: security.capability [ 4.567559] evm: HMAC attrs: 0x1 [ 4.571111] rtc_cmos 00:05: setting system clock to 2025-07-24 13:40:31 UTC (1753364431) [ 4.583338] debug: unmapping init [mem 0xffffffff89603000-0xffffffff897fffff] [ 4.586354] debug: unmapping init [mem 0xffffffff88382000-0xffffffff88658fff] [ 4.594527] Write protecting the kernel read-only data: 28672k [ 4.602705] debug: unmapping init [mem 0xffffffff86a03000-0xffffffff86bfffff] [ 4.610325] debug: unmapping init [mem 0xffffffff87314000-0xffffffff873fffff] [ 4.651544] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.662585] systemd[1]: Detected virtualization kvm. [ 4.664061] systemd[1]: Detected architecture x86-64. [ 4.676407] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.707916] systemd[1]: No hostname configured. [ 4.714187] systemd[1]: Set hostname to . [ 4.716247] random: systemd: uninitialized urandom read (16 bytes read) [ 4.718307] systemd[1]: Initializing machine ID from random generator. [ 4.995694] random: systemd: uninitialized urandom read (16 bytes read) [ 4.998109] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 5.004499] random: systemd: uninitialized urandom read (16 bytes read) [ 5.006754] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 5.014934] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Journal Service... Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.065200] device-mapper: uevent: version 1.0.3 [ 6.070097] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.242202] virtio_net virtio0 ens2: renamed from eth0 [ 7.354012] random: fast init done [ 7.416089] scsi host0: ata_piix [ 7.423845] scsi host1: ata_piix [ 7.425525] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.430677] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.090614] random: crng init done [ 12.092088] random: 7 urandom warning(s) missed due to ratelimiting [ 12.259248] dracut-initqueue[580]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 13.629134] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.336952] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.693903] SELinux: Disabled at runtime. [ 15.764458] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.775233] systemd[1]: Detected virtualization kvm. [ 15.776694] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.603846] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.606614] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.610688] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.620648] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.628030] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.641114] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.650623] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ 16.919429] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 17.220032] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.582738] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 17.612972] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 17.733727] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 17.752104] EDAC sbridge: Ver: 1.1.2 [ 18.642053] hrtimer: interrupt took 2365779 ns [ 20.395917] Key type dns_resolver registered [ 20.879747] NFS: Registering the id_resolver key type [ 20.881528] Key type id_resolver registered [ 20.883890] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg218-server login: [ 53.858651] libcfs: loading out-of-tree module taints kernel. [ 53.867546] Key type ._llcrypt registered [ 53.868347] Key type .llcrypt registered [ 53.904270] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 60.885496] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 61.384482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 61.391387] alg: No test for adler32 (adler32-zlib) [ 62.324442] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 62.587501] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 62.588883] LNet: Accept secure, port 988 [ 64.175107] Key type lgssc registered [ 64.672451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 71.019252] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 71.838377] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 74.982052] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 77.321523] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 79.800348] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 84.691907] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 89.339316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 89.369198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 89.386196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 90.535664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 90.559069] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 90.605795] Lustre: lustre-MDT0000: new disk, initializing [ 90.651155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 90.663280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 93.012919] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 103.344095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 103.414639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 103.486919] Lustre: 6462:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 103.519506] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 103.522064] Lustre: Skipped 1 previous similar message [ 103.577996] Lustre: lustre-MDT0001: new disk, initializing [ 103.678870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 103.705741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 103.712634] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 106.380583] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 110.084697] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 117.289027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 117.342202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 117.482678] Lustre: lustre-OST0000: new disk, initializing [ 117.484977] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 117.538659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 121.585109] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 125.431088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 125.435830] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 125.503656] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 132.569522] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 132.620781] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 132.706846] Lustre: lustre-OST0001: new disk, initializing [ 132.720245] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 132.770668] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 133.999630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 134.004490] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 134.030284] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 136.862780] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 144.897844] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 150.907545] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 158.551397] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing check_logdir /tmp/testlogs/ [ 160.433403] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing yml_node [ 162.272447] Lustre: DEBUG MARKER: Client: 2.16.57.3 [ 163.371404] Lustre: DEBUG MARKER: MDS: 2.16.57.3 [ 164.437210] Lustre: DEBUG MARKER: OSS: 2.16.57.3 [ 165.129201] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jul 24 09:43:11 EDT 2025 [ 172.687961] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 173.284089] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 185.311943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 185.312820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 185.317359] Lustre: Skipped 3 previous similar messages [ 190.432863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 190.437310] Lustre: Skipped 6 previous similar messages [ 191.489050] LustreError: 12122:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 193.504714] LustreError: 12122:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 193.507091] LustreError: 12122:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 193.524269] Lustre: server umount lustre-MDT0000 complete [ 195.393250] LustreError: 10200:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753364622 with bad export cookie 16143097342908682508 [ 195.395482] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 195.402316] LustreError: 10200:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 195.552744] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 195.554982] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 195.556997] Lustre: Skipped 1 previous similar message [ 200.671660] LustreError: 6466:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 200.671662] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 200.671679] LustreError: 6466:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 200.678366] Lustre: Skipped 2 previous similar messages [ 201.539890] LustreError: 12326:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 201.543051] LustreError: 12326:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 201.628536] Lustre: server umount lustre-MDT0001 complete [ 203.271934] Lustre: server umount lustre-OST0000 complete [ 204.783287] LustreError: 12727:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 204.786043] LustreError: 12727:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 204.868904] Lustre: server umount lustre-OST0001 complete [ 207.104189] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 210.941936] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 215.180889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 217.717441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 220.247456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 222.903789] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 225.432389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 225.458252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 225.550880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 225.563362] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 225.601425] Lustre: lustre-MDT0000: new disk, initializing [ 225.630476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 225.638503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 227.090804] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 231.754941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 231.789158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 231.846523] Lustre: 14890:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 231.990633] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 233.813835] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 237.609265] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 239.109576] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 241.870609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 241.904281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 242.031164] Lustre: lustre-OST0000: new disk, initializing [ 242.033233] Lustre: Skipped 1 previous similar message [ 242.035994] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 242.039746] Lustre: Skipped 2 previous similar messages [ 242.075421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 242.078286] Lustre: Skipped 1 previous similar message [ 243.837723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 243.841908] Lustre: Skipped 1 previous similar message [ 243.843906] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 243.886743] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 244.200638] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 247.528238] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 249.277438] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 249.379266] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 251.087285] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 251.197620] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 254.437922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 254.438096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 254.445556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 254.447336] LustreError: Skipped 1 previous similar message [ 254.450064] Lustre: Skipped 1 previous similar message [ 258.159235] LustreError: 17798:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 258.161444] LustreError: 17798:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 258.246355] Lustre: server umount lustre-OST0000 complete [ 262.624858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 262.629162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 262.634680] Lustre: Skipped 1 previous similar message [ 262.638802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 266.087637] Lustre: server umount lustre-MDT0000 complete [ 267.747486] LustreError: 18000:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753364694 with bad export cookie 16143097342908684048 [ 267.751415] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 267.752945] LustreError: 18000:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 267.839505] LustreError: 18201:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 267.843017] LustreError: 18201:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 267.952915] Lustre: server umount lustre-MDT0001 complete [ 270.193108] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 09:44:56 (1753364696) [ 273.673637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 273.886816] LustreError: 18726:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 273.926495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 275.785091] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 279.008632] LustreError: 18726:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 279.734568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 281.686798] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 284.259699] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 285.924748] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 289.161566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 292.206489] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 295.006433] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 306.502329] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 310.753066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 310.756695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 310.762068] Lustre: Skipped 1 previous similar message [ 310.766190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 310.769167] Lustre: Skipped 2 previous similar messages [ 312.392269] LustreError: 21028:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 312.394424] LustreError: 21028:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 312.428580] Lustre: server umount lustre-MDT0000 complete [ 314.187659] LustreError: 18706:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753364741 with bad export cookie 16143097342908684930 [ 314.193313] LustreError: 18706:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 314.193939] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 315.365261] LustreError: 20865:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.425730] Lustre: server umount lustre-MDT0001 complete [ 318.312434] Lustre: server umount lustre-OST0000 complete [ 327.845952] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 333.381381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 333.630264] LustreError: 22744:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 333.677323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 333.680735] Lustre: Skipped 2 previous similar messages [ 335.638928] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 339.761221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 341.716597] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 343.049263] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 346.202136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 348.792822] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 352.649966] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 352.684853] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 352.764348] Lustre: lustre-OST0001: new disk, initializing [ 352.766839] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 354.384247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 354.388339] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 354.438181] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 355.189603] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 360.362652] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 362.075582] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 375.269519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 375.271815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 375.275501] Lustre: Skipped 4 previous similar messages [ 375.278479] Lustre: Skipped 2 previous similar messages [ 378.574189] LustreError: 26199:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 378.577514] LustreError: 26199:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 378.622650] Lustre: server umount lustre-MDT0000 complete [ 380.334859] LustreError: 25944:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753364807 with bad export cookie 16143097342908685770 [ 380.340446] LustreError: 25944:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 380.341158] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 380.386093] LustreError: 22740:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 380.396117] LustreError: 22740:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 380.565876] Lustre: server umount lustre-MDT0001 complete [ 388.561265] Lustre: server umount lustre-OST0000 complete [ 390.424562] Lustre: server umount lustre-OST0001 complete [ 397.307762] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 402.136347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 402.366425] LustreError: 27964:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 402.422224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 402.425113] Lustre: Skipped 3 previous similar messages [ 404.241579] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 408.109422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 409.964231] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 411.176028] Lustre: 29068:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 413.823801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 416.197667] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 419.964840] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 422.796350] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 431.977246] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 433.526398] Lustre: 30909:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 442.850186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 442.850907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 442.854431] Lustre: Skipped 5 previous similar messages [ 442.856733] Lustre: Skipped 5 previous similar messages [ 446.593808] LustreError: 31110:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 446.596489] LustreError: 31110:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 446.627526] Lustre: server umount lustre-MDT0000 complete [ 447.970231] LustreError: 28587:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 447.977131] LustreError: 28587:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 448.030587] LustreError: 27944:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753364874 with bad export cookie 16143097342908687072 [ 448.035285] LustreError: 27944:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 448.035679] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 448.193158] Lustre: server umount lustre-MDT0001 complete [ 459.277643] Lustre: server umount lustre-OST0000 complete [ 470.366106] Lustre: server umount lustre-OST0001 complete [ 474.148603] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 09:48:20 (1753364900) [ 485.009049] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 489.022313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 490.688907] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 493.991593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 495.787806] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 496.927463] Lustre: 34783:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 499.524838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 502.005583] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 505.300926] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 507.489548] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 522.657869] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 525.569905] Lustre: 36643:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 535.008046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 535.012316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 535.024223] Lustre: Skipped 1 previous similar message [ 541.090591] Lustre: server umount lustre-MDT0000 complete [ 544.801655] LustreError: 33659:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753364971 with bad export cookie 16143097342908688178 [ 544.812449] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 544.820597] LustreError: 33659:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 545.259285] Lustre: server umount lustre-MDT0001 complete [ 559.951526] Lustre: server umount lustre-OST0000 complete [ 561.119254] Lustre: 3635:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753364972/real 1753364972] req@ffff9c0442c30000 x1838535922697984/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1753364988 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 562.107349] Lustre: server umount lustre-OST0001 complete [ 567.007242] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 09:49:53 (1753364993) [ 573.885330] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 578.564089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 578.774846] LustreError: 38754:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 578.788191] LustreError: 38754:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 578.843093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 578.848110] Lustre: Skipped 7 previous similar messages [ 580.808352] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 585.156902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 587.007788] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 588.261466] Lustre: 39856:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 591.083287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 593.797800] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 598.061862] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 601.040658] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 610.450896] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 617.755774] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 617.758749] Lustre: Skipped 1 previous similar message [ 619.489704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 619.493301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 619.498481] Lustre: Skipped 6 previous similar messages [ 619.501390] Lustre: Skipped 11 previous similar messages [ 625.292798] LustreError: 41909:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 625.296810] LustreError: 41909:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 625.312616] LustreError: 41909:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff9c05432bdd00) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 627.379151] Lustre: server umount lustre-MDT0000 complete [ 629.175463] LustreError: 38735:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753365056 with bad export cookie 16143097342908689284 [ 629.180703] LustreError: 38735:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 629.181088] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 635.458692] Lustre: server umount lustre-MDT0001 complete [ 640.440567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 656.776472] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 661.545808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 663.711554] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 664.874532] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 666.343585] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 669.289501] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 671.092144] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 674.086269] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 677.344987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 682.623332] Lustre: server umount lustre-MDT0000 complete [ 684.525344] LustreError: 38735:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753365111 with bad export cookie 16143097342908690474 [ 684.529231] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 684.530191] LustreError: 38735:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 684.717302] Lustre: server umount lustre-MDT0001 complete [ 696.781654] Lustre: server umount lustre-OST0000 complete [ 708.810910] Lustre: server umount lustre-OST0001 complete [ 712.379769] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 09:52:19 (1753365139) [ 718.699920] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 722.738338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 722.914217] LustreError: 46059:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 722.924460] LustreError: 46059:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 12 previous similar messages [ 724.636956] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 728.498080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 730.515493] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 731.838094] Lustre: 47165:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 731.842625] Lustre: 47165:0:(mgs_llog.c:1345:mgs_modify_param()) Skipped 1 previous similar message [ 734.932613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 737.642786] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 742.093390] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 744.841643] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 754.060365] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 761.107545] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 767.969312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 767.974668] Lustre: Skipped 10 previous similar messages [ 772.573930] Lustre: server umount lustre-MDT0000 complete [ 774.031389] LustreError: 46039:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753365200 with bad export cookie 16143097342908691258 [ 774.034159] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 774.035798] LustreError: 46039:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 774.183505] Lustre: server umount lustre-MDT0001 complete [ 785.412507] Lustre: server umount lustre-OST0000 complete [ 797.184739] Lustre: server umount lustre-OST0001 complete [ 801.141641] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 09:53:47 (1753365227) [ 802.251644] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 802.964700] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 09:53:49 (1753365229) [ 807.110713] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 810.404312] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 814.556800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 817.107273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 819.950941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 819.984690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 820.105981] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 820.147632] Lustre: lustre-MDT0000: new disk, initializing [ 820.194771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 821.845130] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 825.510459] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 828.171169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 828.196781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 829.406688] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 829.424327] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 830.733838] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 834.378730] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 835.256202] Lustre: DEBUG MARKER: create test files [ 849.889543] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 853.761178] Lustre: server umount lustre-OST0000 complete [ 855.437343] Lustre: server umount lustre-MDT0000 complete [ 860.756346] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 862.038177] Key type lgssc unregistered [ 862.204207] LNet: 54526:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.268835] LNet: Removed LNI 192.168.202.118@tcp [ 863.631668] Key type .llcrypt unregistered [ 863.633423] Key type ._llcrypt unregistered [ 875.983110] Key type ._llcrypt registered [ 875.984795] Key type .llcrypt registered [ 876.035166] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 876.387962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 876.422101] alg: No test for adler32 (adler32-zlib) [ 877.307801] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 877.404921] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 877.407509] LNet: Accept secure, port 988 [ 878.999749] Key type lgssc registered [ 879.407154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 882.553504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 882.565103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 883.783107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 885.271864] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 887.400887] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 889.919876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 890.062768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 892.254541] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 893.156696] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 893.186038] Lustre: 57090:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 893.190287] Lustre: 57090:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 893.192982] Lustre: 57090:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 893.196276] Lustre: 57090:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 893.199818] Lustre: 57090:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 893.202615] Lustre: 57090:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 894.346942] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 897.603626] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 899.993864] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 903.648878] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 903.648958] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 903.657372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 908.334130] LustreError: 57828:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 908.405622] Lustre: server umount lustre-OST0000 complete [ 910.017102] LustreError: 58030:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 910.019850] LustreError: 58030:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 910.110495] Lustre: server umount lustre-MDT0000 complete [ 915.414339] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 916.640208] Key type lgssc unregistered [ 916.805511] LNet: 58557:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 917.861675] LNet: Removed LNI 192.168.202.118@tcp [ 918.201990] Key type .llcrypt unregistered [ 918.203560] Key type ._llcrypt unregistered [ 930.118992] Key type ._llcrypt registered [ 930.120598] Key type .llcrypt registered [ 930.160667] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 930.486039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 930.493833] alg: No test for adler32 (adler32-zlib) [ 931.362552] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 931.452039] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 931.454679] LNet: Accept secure, port 988 [ 933.039188] Key type lgssc registered [ 933.426446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 936.447622] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 936.459132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 937.662545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 939.112499] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 941.300496] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 944.000207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 944.128336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 946.343619] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 947.236858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 947.261083] Lustre: 61220:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 947.264828] Lustre: 61220:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 947.267527] Lustre: 61220:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 947.270201] Lustre: 61220:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 947.272733] Lustre: 61220:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 947.274837] Lustre: 61220:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 948.417568] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 951.560722] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 956.335087] LustreError: 61903:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 956.395656] Lustre: server umount lustre-OST0000 complete [ 967.139236] LustreError: 62105:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 967.141865] LustreError: 62105:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 967.249867] Lustre: server umount lustre-MDT0000 complete [ 972.407620] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 973.636677] Key type lgssc unregistered [ 973.781482] LNet: 62635:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 974.821429] LNet: Removed LNI 192.168.202.118@tcp [ 975.165980] Key type .llcrypt unregistered [ 975.167699] Key type ._llcrypt unregistered [ 982.297435] Key type ._llcrypt registered [ 982.298783] Key type .llcrypt registered [ 982.342723] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 989.349961] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 989.846222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 989.877406] alg: No test for adler32 (adler32-zlib) [ 990.765171] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 990.866816] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 990.869659] LNet: Accept secure, port 988 [ 992.455176] Key type lgssc registered [ 992.898159] Lustre: Echo OBD driver; http://www.lustre.org/ [ 996.420245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 998.939317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1001.115467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1003.342080] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1005.811499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1005.830555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1005.837178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1006.928245] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1006.941724] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1006.981620] Lustre: lustre-MDT0000: new disk, initializing [ 1007.015129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1007.024359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1008.485564] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1012.903606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1012.926858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1012.962321] Lustre: 65721:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1012.987602] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1012.989699] Lustre: Skipped 1 previous similar message [ 1013.027782] Lustre: lustre-MDT0001: new disk, initializing [ 1013.061173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1013.072756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1013.077395] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1014.491375] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1017.864520] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1019.194228] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1021.891785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1021.925268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1022.048613] Lustre: lustre-OST0000: new disk, initializing [ 1022.051858] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1022.096540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1024.415833] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1027.839378] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1029.416419] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1031.686791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1031.692804] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1031.738820] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1032.592886] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 1034.199175] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1034.293575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1036.774128] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1036.774882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1036.777826] LustreError: Skipped 1 previous similar message [ 1036.779816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1036.783023] Lustre: Skipped 1 previous similar message [ 1036.789876] Lustre: Skipped 1 previous similar message [ 1041.132127] LustreError: 68651:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1041.204539] Lustre: server umount lustre-OST0000 complete [ 1043.936679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1043.940853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1043.947649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1047.008179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1047.008400] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1052.128309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1052.131210] Lustre: Skipped 2 previous similar messages [ 1056.224406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1056.226449] Lustre: Skipped 1 previous similar message [ 1057.247189] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1057.285950] LustreError: 68852:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1057.288677] LustreError: 68852:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1057.317964] Lustre: server umount lustre-MDT0000 complete [ 1058.762148] LustreError: 65712:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753365485 with bad export cookie 673266565680662301 [ 1058.765381] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1058.767749] LustreError: 65712:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1058.834217] LustreError: 69055:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1058.836932] LustreError: 69055:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1058.921521] Lustre: server umount lustre-MDT0001 complete [ 1062.163993] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 09:58:09 (1753365489) [ 1066.332434] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 1071.925757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1081.540138] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 1084.960541] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 1088.706087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1091.288580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1093.569061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1095.972629] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1098.580585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1098.605527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1098.696756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1098.712079] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1098.753219] Lustre: lustre-MDT0000: new disk, initializing [ 1098.788992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1098.797465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1100.309325] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1104.852278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1104.877211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1104.915112] Lustre: 73258:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1104.940243] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1104.942826] Lustre: Skipped 1 previous similar message [ 1104.980697] Lustre: lustre-MDT0001: new disk, initializing [ 1105.016927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1105.030682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1105.035077] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1106.496293] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1109.878152] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1111.375635] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1114.085325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1114.119296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1114.232899] Lustre: lustre-OST0000: new disk, initializing [ 1114.235745] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1114.268875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1116.474662] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1119.708112] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1120.771974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1120.775819] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1120.825146] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1121.274663] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1121.363594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1122.890330] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1122.978567] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1125.857111] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1125.857632] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1125.860530] LustreError: Skipped 1 previous similar message [ 1125.866782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1125.870629] Lustre: Skipped 1 previous similar message [ 1129.774038] LustreError: 76167:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1129.777069] LustreError: 76167:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1129.842650] Lustre: server umount lustre-OST0000 complete [ 1136.096660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1136.097170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1136.105550] Lustre: Skipped 1 previous similar message [ 1137.408680] LustreError: 76368:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1137.411121] LustreError: 76368:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1137.440625] Lustre: server umount lustre-MDT0000 complete [ 1138.807214] LustreError: 73251:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753365565 with bad export cookie 673266565680663169 [ 1138.809533] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1138.812292] LustreError: 73251:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1138.965717] Lustre: server umount lustre-MDT0001 complete [ 1141.962665] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 09:59:28 (1753365568) [ 1144.352029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1144.546761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1146.022804] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1149.283996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1150.913077] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1152.975662] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1154.400207] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1156.899454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1159.139644] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1161.179813] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1162.493301] Lustre: 79558:0:(genops.c:1791:obd_export_evict_by_uuid()) MGS: evicting 662be630-4745-4f8a-abb7-d6b999e64f68 at adminstrative request [ 1164.256883] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1164.261476] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 1174.500166] Lustre: MGS: Client 662be630-4745-4f8a-abb7-d6b999e64f68 (at 0@lo) reconnecting [ 1174.503048] Lustre: Evicted from MGS (at 192.168.202.118@tcp) after server handle changed from 0x957ec5fd68b0de5 to 0x957ec5fd68b1070 [ 1174.509288] Lustre: MGC192.168.202.118@tcp: Connection restored to 192.168.202.118@tcp (at 0@lo) [ 1195.291984] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1195.325899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1195.391463] Lustre: lustre-OST0001: new disk, initializing [ 1195.394334] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1195.436079] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1195.438729] Lustre: Skipped 2 previous similar messages [ 1197.589169] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1200.882899] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1201.885124] LustreError: 80675:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 1201.887425] LustreError: 80675:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1201.903770] Lustre: server umount lustre-OST0001 complete [ 1204.717913] LustreError: 78823:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1204.721090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1204.722539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1204.722547] Lustre: Skipped 2 previous similar messages [ 1204.724768] LustreError: 78823:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1204.729503] Lustre: Skipped 2 previous similar messages [ 1209.317507] Lustre: server umount lustre-OST0000 complete [ 1210.847896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1225.183160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1225.219238] LustreError: 81085:0:(obd_class.h:479:obd_check_dev()) Device 24 not setup [ 1225.221920] LustreError: 81085:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1225.252322] Lustre: server umount lustre-MDT0000 complete [ 1226.584163] LustreError: 77219:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753365653 with bad export cookie 673266565680664688 [ 1226.587765] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1226.592359] LustreError: 77219:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1226.732131] Lustre: server umount lustre-MDT0001 complete [ 1229.543841] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 10:00:56 (1753365656) [ 1230.066103] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 1230.689600] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 10:00:57 (1753365657) [ 1231.234600] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 1231.817625] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 10:00:58 (1753365658) [ 1232.331358] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 1232.888966] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 10:00:59 (1753365659) [ 1233.405043] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 1235.115254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1235.463744] loop: module loaded [ 1235.467887] loop0: detected capacity change from 0 to 4294967296 [ 1235.484743] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 1236.023825] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 1237.052224] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 1238.505912] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 1241.511194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1242.822473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1244.133541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1245.446395] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1247.781981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1247.808782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1247.821093] LustreError: 83863:0:(obd_config.c:779:class_setup()) setup lustre-OST0000-osd failed (-28) [ 1247.824634] LustreError: 83863:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 1247.827910] LustreError: 83863:0:(tgt_mount.c:2240:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 1247.831535] LustreError: 83863:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 1254.262645] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 1257.672687] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 1261.453591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1263.880694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1266.157158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1268.527447] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1271.081821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1271.109576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1271.208345] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1271.222701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1271.261453] Lustre: lustre-MDT0000: new disk, initializing [ 1271.292893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1271.301084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1272.805262] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1277.415742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1277.443494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1277.478793] Lustre: 86684:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1277.584321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1279.016688] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1282.228573] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1283.452389] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1285.887276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1285.911415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1288.210816] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1291.810145] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1291.861295] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1291.949559] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1295.273796] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1295.470093] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1299.412176] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1299.668852] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1301.990704] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1301.994762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1302.008637] LustreError: Skipped 1 previous similar message [ 1302.009721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1302.009725] Lustre: Skipped 8 previous similar messages [ 1302.061607] Lustre: Skipped 3 previous similar messages [ 1316.319346] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1316.361857] LustreError: 89594:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1316.369504] LustreError: 89594:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 1316.593984] Lustre: server umount lustre-OST0000 complete [ 1326.810181] Lustre: server umount lustre-MDT0000 complete [ 1327.589466] LustreError: 86695:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1327.626743] LustreError: 86695:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1330.820970] LustreError: 88419:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753365757 with bad export cookie 673266565680665248 [ 1330.825630] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1330.949625] LustreError: 88419:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1331.344572] Lustre: server umount lustre-MDT0001 complete [ 1341.149884] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 10:02:47 (1753365767) [ 1351.244555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1352.156041] LustreError: 90672:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1357.281660] LustreError: 90672:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1358.193953] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1362.407636] LustreError: 90672:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1371.627898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1378.365050] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1385.571821] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1390.162465] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1397.137847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1402.774043] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1407.994656] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1411.606340] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1423.485669] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 11 sec [ 1426.443996] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1426.596809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1431.760702] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1431.813696] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1431.940339] Lustre: lustre-OST0001: new disk, initializing [ 1431.955932] Lustre: Skipped 2 previous similar messages [ 1431.967880] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1431.979045] Lustre: Skipped 3 previous similar messages [ 1432.065455] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1432.068081] Lustre: Skipped 5 previous similar messages [ 1437.044221] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1437.270102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1437.275096] Lustre: Skipped 2 previous similar messages [ 1437.278950] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1437.346234] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1442.416922] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1445.631309] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1445.800863] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1449.266964] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1449.441974] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1466.040255] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1494.379815] Lustre: Failing over lustre-MDT0000 [ 1494.781736] LustreError: 95613:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1494.797390] LustreError: 95613:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 1495.143326] Lustre: server umount lustre-MDT0000 complete [ 1495.521502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1495.534107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1495.553787] Lustre: Skipped 2 previous similar messages [ 1495.564679] LustreError: 90672:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1495.585186] LustreError: 90672:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1513.063107] LustreError: 90667:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1513.090413] LustreError: 90667:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 19 previous similar messages [ 1515.493353] Lustre: 63942:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753365926/real 1753365926] req@00000000818a2c59 x1838536895715456/t0(0) o400->MGC192.168.202.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1753365942 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1515.524654] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1517.489278] LDISKFS-fs (dm-0): recovery complete [ 1517.493216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1525.969305] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1528.809553] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1529.449304] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 1529.452945] Lustre: 90666:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1531.364929] Lustre: 91565:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1531.379438] Lustre: 91565:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 1 previous similar message [ 1531.391583] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1531.448269] LustreError: 96361:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1531.768040] LustreError: 96361:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1532.091200] LustreError: 96361:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1532.100879] LustreError: 96361:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 1 previous similar message [ 1532.424324] LustreError: 96361:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1532.442036] LustreError: 96361:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 1 previous similar message [ 1532.455047] Lustre: 96361:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1532.472218] Lustre: 96361:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 7 previous similar messages [ 1533.125520] LustreError: 96361:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1533.139109] LustreError: 96361:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 2 previous similar messages [ 1533.471693] LustreError: 96361:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1533.488482] LustreError: 96361:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 2 previous similar messages [ 1534.761825] Lustre: 96361:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1534.784918] Lustre: 96361:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 6 previous similar messages [ 1535.419466] LustreError: 96361:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1535.427405] LustreError: 96361:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 6 previous similar messages [ 1535.751155] LustreError: 96361:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1535.754063] LustreError: 96361:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 6 previous similar messages [ 1538.927924] Lustre: 96361:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1538.939828] Lustre: 96361:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 12 previous similar messages [ 1539.575910] LustreError: 96361:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1539.581168] LustreError: 96361:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 12 previous similar messages [ 1539.895621] LustreError: 96361:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1539.899440] LustreError: 96361:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 12 previous similar messages [ 1547.187083] Lustre: 96361:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1547.199922] Lustre: 96361:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 25 previous similar messages [ 1547.844360] LustreError: 96361:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1547.856187] LustreError: 96361:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 25 previous similar messages [ 1548.175129] LustreError: 96361:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1548.184351] LustreError: 96361:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 25 previous similar messages [ 1563.353590] Lustre: 96361:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1563.369414] Lustre: 96361:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 50 previous similar messages [ 1564.018168] LustreError: 96361:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1564.022148] LustreError: 96361:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 50 previous similar messages [ 1564.343151] LustreError: 96361:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1564.351572] LustreError: 96361:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 50 previous similar messages [ 1589.215213] Lustre: 96361:0:(ldlm_lib.c:2309:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 1589.223844] Lustre: 96361:0:(ldlm_lib.c:2319:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1589.228879] Lustre: 96361:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 47be8dbd-af9a-4c76-961f-8a3802c1afcb@192.168.202.18@tcp [ 1589.234606] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1589.237591] Lustre: 96361:0:(ldlm_lib.c:1831:abort_req_replay_queue()) @@@ aborted: req@00000000927708b4 x1838536889275392/t0(8589934778) o35->47be8dbd-af9a-4c76-961f-8a3802c1afcb@192.168.202.18@tcp:81/0 lens 392/0 e 0 to 0 dl 1753366026 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 1589.252363] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1589.268642] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 1589.280224] Lustre: lustre-MDT0000: Denying connection for new client 47be8dbd-af9a-4c76-961f-8a3802c1afcb (at 192.168.202.18@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) already passed deadline 0:00 [ 1589.301788] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 1589.305779] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 1589.306144] Lustre: Skipped 2 previous similar messages [ 1589.368701] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:48 to 0x2c0000401:545) [ 1589.368756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:79 to 0x280000401:577) [ 1605.610276] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1605.613893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1605.626973] Lustre: Skipped 3 previous similar messages [ 1605.631486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1605.641250] Lustre: Skipped 7 previous similar messages [ 1610.333246] Lustre: server umount lustre-OST0000 complete [ 1613.285172] LustreError: 92260:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1613.308211] LustreError: 92260:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 23 previous similar messages [ 1619.787646] Lustre: server umount lustre-OST0001 complete [ 1636.319164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1636.392598] LustreError: 97421:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 1636.410695] LustreError: 97421:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 1636.463280] Lustre: server umount lustre-MDT0000 complete [ 1639.075581] LustreError: 93728:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753366066 with bad export cookie 673266565680716453 [ 1639.080015] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1639.081104] LustreError: 93728:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1639.332296] Lustre: server umount lustre-MDT0001 complete [ 1644.984759] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 10:07:51 (1753366071) [ 1649.522532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1649.526863] Lustre: *** cfs_fail_loc=197, val=0*** [ 1666.015746] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1672.165471] LustreError: 98316:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.118@tcp: failed processing log, type 4: rc = -110 [ 1678.315427] Lustre: *** cfs_fail_loc=197, val=0*** [ 1678.317131] Lustre: Skipped 65 previous similar messages [ 1701.857408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1701.860152] Lustre: Skipped 1 previous similar message [ 1704.290944] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1706.481761] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1707.364361] Lustre: server umount lustre-OST0000 complete [ 1710.822534] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 10:08:57 (1753366137) [ 1725.532804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1725.766768] LustreError: 99818:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1725.775046] LustreError: 99818:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 1727.402567] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1730.950304] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1733.137349] LustreError: 99849:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1733.142810] LustreError: 99849:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1733.147655] LustreError: 99849:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 1733.337730] Lustre: server umount lustre-MDT0000 complete [ 1740.128894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1746.919511] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 1750.234185] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 1754.209046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1756.964510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1759.607594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1762.235963] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1764.886509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1764.911793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1765.003437] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1765.017606] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1765.057863] Lustre: lustre-MDT0000: new disk, initializing [ 1765.095646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1766.477569] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1770.918716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1770.945193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1770.987625] Lustre: 103859:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1771.096420] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1772.569610] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1775.922546] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1777.280176] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1779.782960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1779.810070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1782.090983] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1785.397442] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1786.894369] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1788.969351] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1789.037793] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 1790.348928] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1790.424771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1794.017596] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1794.017934] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1794.019465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1794.019469] Lustre: Skipped 13 previous similar messages [ 1794.020484] LustreError: Skipped 4 previous similar messages [ 1794.030127] Lustre: Skipped 6 previous similar messages [ 1797.226406] Lustre: server umount lustre-OST0000 complete [ 1804.630441] Lustre: server umount lustre-MDT0000 complete [ 1806.022492] LustreError: 103849:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753366232 with bad export cookie 673266565680727107 [ 1806.025579] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1806.026462] LustreError: 103849:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1806.193823] Lustre: server umount lustre-MDT0001 complete [ 1809.779832] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 10:10:36 (1753366236) [ 1814.151352] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 1815.297756] Key type lgssc unregistered [ 1815.448420] LNet: 107902:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1816.485422] LNet: Removed LNI 192.168.202.118@tcp [ 1816.786331] Key type .llcrypt unregistered [ 1816.787903] Key type ._llcrypt unregistered [ 1822.981489] Key type ._llcrypt registered [ 1822.982559] Key type .llcrypt registered [ 1823.020709] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 1829.398279] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 1829.812041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1829.818544] alg: No test for adler32 (adler32-zlib) [ 1830.678450] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 1830.764089] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 1830.766523] LNet: Accept secure, port 988 [ 1832.351253] Key type lgssc registered [ 1832.773124] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1836.142490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1838.853689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1841.110382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1843.363741] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1845.527505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1848.057430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1848.072584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1848.077852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1849.165890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1849.179219] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1849.218177] Lustre: lustre-MDT0000: new disk, initializing [ 1849.244336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1849.251460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1850.659625] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1855.122526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1855.151135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1855.255314] Lustre: lustre-OST0000: new disk, initializing [ 1855.258276] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1855.260438] Lustre: Skipped 1 previous similar message [ 1855.285942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1857.032351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1857.036234] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1857.049637] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1857.420632] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1862.054728] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1862.079335] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1862.140386] Lustre: lustre-OST0001: new disk, initializing [ 1862.143097] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1862.174062] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1864.342292] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1872.352691] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1872.353217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1872.357847] Lustre: Skipped 1 previous similar message [ 1872.360612] Lustre: Skipped 1 previous similar message [ 1874.814431] LustreError: 113414:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1874.910873] Lustre: server umount lustre-MDT0000 complete [ 1877.245049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1878.958987] LustreError: 111194:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753366305 with bad export cookie 1942030342160374083 [ 1878.962055] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1878.962918] LustreError: 111194:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1878.981088] LustreError: 114009:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 1878.982855] LustreError: 114009:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1878.998800] Lustre: server umount lustre-OST0000 complete [ 1880.297474] LustreError: 114209:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1880.300445] LustreError: 114209:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1880.366144] Lustre: server umount lustre-OST0001 complete [ 1883.594847] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 10:11:50 (1753366310) [ 1887.289043] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 1890.730212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1891.172189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1893.229465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1893.620958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1896.099527] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 10:12:03 (1753366323) [ 1900.761987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1901.486661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1902.185351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1902.918916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1903.300930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1904.013514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1910.215246] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 1913.884593] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 1917.524463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1920.188796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1922.582441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1925.352534] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1928.015529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1928.036865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1928.131229] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1928.144433] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1928.183480] Lustre: lustre-MDT0000: new disk, initializing [ 1928.216193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1928.225348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1929.629479] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1933.911600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1933.935695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1933.963927] Lustre: 119500:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1933.986227] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1933.987881] Lustre: Skipped 1 previous similar message [ 1934.023913] Lustre: lustre-MDT0001: new disk, initializing [ 1934.056249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1934.069266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1934.072623] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1935.309866] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1938.501030] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1939.742135] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1942.148684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1942.173453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1942.265583] Lustre: lustre-OST0000: new disk, initializing [ 1942.268179] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1942.298575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1944.202443] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1946.614047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1946.617902] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1946.647182] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1947.236700] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1948.644053] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1948.720853] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1950.209584] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1950.306482] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1951.714446] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1951.715110] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1951.715597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1951.718729] Lustre: Skipped 1 previous similar message [ 1951.721565] LustreError: Skipped 1 previous similar message [ 1953.068773] LustreError: 122407:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1953.071396] LustreError: 122407:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1953.134292] Lustre: server umount lustre-OST0000 complete [ 1954.783775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1954.787675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1954.791785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1954.794740] Lustre: Skipped 1 previous similar message [ 1956.832625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1956.832740] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1960.506605] LustreError: 122609:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1960.508466] LustreError: 122609:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1960.539567] Lustre: server umount lustre-MDT0000 complete [ 1961.835250] LustreError: 119491:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753366388 with bad export cookie 1942030342160375224 [ 1961.838171] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1961.839615] LustreError: 119491:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1961.989338] Lustre: server umount lustre-MDT0001 complete [ 1965.065114] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 10:13:12 (1753366392) [ 1967.155023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1967.318514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1968.521082] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1971.336504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1972.756421] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1974.697967] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1975.995888] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1978.303532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1980.370224] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1982.207874] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1994.906500] Lustre: *** cfs_fail_loc=159, val=0*** [ 1994.908337] Lustre: Skipped 1 previous similar message [ 2011.250078] Lustre: lustre-MDT0000: Client a876b056-8105-4434-a94e-086764b487c0 (at 192.168.202.18@tcp) reconnecting [ 2011.771619] Lustre: *** cfs_fail_loc=159, val=0*** [ 2011.773469] Lustre: Skipped 4 previous similar messages [ 2028.140983] Lustre: lustre-MDT0000: Client a876b056-8105-4434-a94e-086764b487c0 (at 192.168.202.18@tcp) reconnecting [ 2032.607854] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2032.610186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2032.614567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2032.616103] Lustre: Skipped 1 previous similar message [ 2035.436069] LustreError: 125966:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2035.438233] LustreError: 125966:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2035.490989] Lustre: server umount lustre-OST0000 complete [ 2038.239893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2038.242156] LustreError: Skipped 1 previous similar message [ 2042.834542] Lustre: server umount lustre-MDT0000 complete [ 2043.872156] LustreError: 124186:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2043.875933] LustreError: 124186:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2044.003940] LustreError: 123460:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753366470 with bad export cookie 1942030342160376092 [ 2044.006696] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2044.007365] LustreError: 123460:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2044.136423] Lustre: server umount lustre-MDT0001 complete [ 2048.674960] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 2049.732595] Key type lgssc unregistered [ 2049.881415] LNet: 126896:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2050.920325] LNet: Removed LNI 192.168.202.118@tcp [ 2051.204785] Key type .llcrypt unregistered [ 2051.205732] Key type ._llcrypt unregistered [ 2059.854137] Key type ._llcrypt registered [ 2059.855091] Key type .llcrypt registered [ 2059.892191] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 2060.182458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2060.189711] alg: No test for adler32 (adler32-zlib) [ 2061.031725] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 2061.109739] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 2061.111538] LNet: Accept secure, port 988 [ 2062.687155] Key type lgssc registered [ 2063.025466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2065.522108] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2065.529472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2066.682831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2068.028622] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2070.830543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2070.945754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2072.210207] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2074.027623] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2075.225993] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2077.433884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2077.540064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2079.468272] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2081.295970] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2092.901526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 2101.298126] Lustre: *** cfs_fail_loc=159, val=0*** [ 2116.716626] Lustre: lustre-MDT0000: Client be0e2d24-825d-4621-93b4-e2c3b8371df2 (at 192.168.202.18@tcp) reconnecting [ 2117.428641] Lustre: *** cfs_fail_loc=159, val=0*** [ 2117.430121] Lustre: Skipped 1 previous similar message [ 2132.589365] Lustre: lustre-MDT0001: Client be0e2d24-825d-4621-93b4-e2c3b8371df2 (at 192.168.202.18@tcp) reconnecting [ 2133.093821] Lustre: *** cfs_fail_loc=159, val=0*** [ 2133.095598] Lustre: Skipped 1 previous similar message [ 2149.488431] Lustre: lustre-MDT0001: Client be0e2d24-825d-4621-93b4-e2c3b8371df2 (at 192.168.202.18@tcp) reconnecting [ 2150.644749] Lustre: 130964:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2151.490610] Lustre: *** cfs_fail_loc=159, val=0*** [ 2151.492044] Lustre: Skipped 4 previous similar messages [ 2166.893332] Lustre: lustre-MDT0000: Client e26642d3-88db-4352-82d0-4584bcbc22f0 (at 192.168.202.18@tcp) reconnecting [ 2167.495232] Lustre: *** cfs_fail_loc=159, val=0*** [ 2167.498884] Lustre: Skipped 13 previous similar messages [ 2182.769116] Lustre: lustre-MDT0000: Client e26642d3-88db-4352-82d0-4584bcbc22f0 (at 192.168.202.18@tcp) reconnecting [ 2183.306329] Lustre: 131251:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2185.187167] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2185.187843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2185.189516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2185.191218] Lustre: Skipped 1 previous similar message [ 2190.304663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2190.307204] Lustre: Skipped 2 previous similar messages [ 2190.700114] LustreError: 131349:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2190.762532] Lustre: server umount lustre-OST0000 complete [ 2193.888161] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2193.892945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2193.899863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2195.425016] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2198.206218] LustreError: 131551:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2198.209224] LustreError: 131551:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2198.236964] Lustre: server umount lustre-MDT0000 complete [ 2199.600229] LustreError: 128237:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753366626 with bad export cookie 8768901447031250604 [ 2199.603885] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2199.604957] LustreError: 128237:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2199.665346] LustreError: 131752:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2199.667769] LustreError: 131752:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2199.747862] Lustre: server umount lustre-MDT0001 complete [ 2205.007736] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 2206.944600] Key type lgssc unregistered [ 2207.210770] LNet: 132278:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2208.232584] LNet: Removed LNI 192.168.202.118@tcp [ 2209.066550] Key type .llcrypt unregistered [ 2209.069103] Key type ._llcrypt unregistered [ 2229.127686] Key type ._llcrypt registered [ 2229.129459] Key type .llcrypt registered [ 2229.204175] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 2230.089944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2230.138230] alg: No test for adler32 (adler32-zlib) [ 2231.178244] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 2231.362455] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 2231.365615] LNet: Accept secure, port 988 [ 2232.992352] Key type lgssc registered [ 2233.876731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2239.675777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2239.694851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2241.150182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2244.154789] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2251.760368] LustreError: 133656:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2251.777558] LustreError: 133656:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2253.079968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2253.327424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2257.007601] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2262.659756] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2266.209921] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2271.187980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2271.412900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2274.677905] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2278.033402] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2278.576320] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 2286.772190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 2292.990867] Lustre: 135912:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2294.357909] Lustre: 135960:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2295.118166] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 2297.170331] Lustre: 136107:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 2297.655715] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 2297.660084] Lustre: Skipped 1 previous similar message [ 2298.137164] Lustre: 136205:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2302.432981] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2302.437834] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2302.440203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2302.444225] Lustre: Skipped 1 previous similar message [ 2306.029306] LustreError: 136303:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2306.123739] Lustre: server umount lustre-OST0000 complete [ 2309.603043] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2309.606769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2309.612379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2309.615480] Lustre: Skipped 1 previous similar message [ 2312.674288] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2312.675160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2312.686814] Lustre: Skipped 1 previous similar message [ 2317.793433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2317.796087] Lustre: Skipped 1 previous similar message [ 2322.399129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2322.453438] LustreError: 136511:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2322.461889] LustreError: 136511:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2322.508774] Lustre: server umount lustre-MDT0000 complete [ 2322.912825] LustreError: 133656:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2322.919669] LustreError: 133656:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2324.640132] LustreError: 136512:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753366751 with bad export cookie 13656448711990241580 [ 2324.643347] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2324.645675] LustreError: 136512:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2324.739273] LustreError: 136714:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2324.742598] LustreError: 136714:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2324.877874] Lustre: server umount lustre-MDT0001 complete [ 2331.473184] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 2333.077892] Key type lgssc unregistered [ 2333.233931] LNet: 137240:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2334.249550] LNet: Removed LNI 192.168.202.118@tcp [ 2334.823176] Key type .llcrypt unregistered [ 2334.825059] Key type ._llcrypt unregistered [ 2347.480831] Key type ._llcrypt registered [ 2347.482400] Key type .llcrypt registered [ 2347.536177] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 2347.964291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2348.007222] alg: No test for adler32 (adler32-zlib) [ 2348.888293] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 2349.014775] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 2349.017399] LNet: Accept secure, port 988 [ 2350.623282] Key type lgssc registered [ 2351.151535] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2355.002930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2355.030389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2356.324712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2358.222216] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2362.214574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2362.436531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2364.541618] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2367.497266] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2369.283453] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2372.961442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2373.171994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2376.270466] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2379.657740] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2381.356229] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 2387.508907] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 2392.420707] Lustre: *** cfs_fail_loc=159, val=0*** [ 2408.066921] Lustre: lustre-MDT0000: Client 9e8ad2a7-8b88-486e-b7db-bc7fd362cafa (at 192.168.202.18@tcp) reconnecting [ 2412.000940] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2412.004884] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2412.014567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2413.035158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2413.036119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2413.042737] Lustre: Skipped 1 previous similar message [ 2416.820602] LustreError: 140981:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2416.941763] Lustre: server umount lustre-OST0000 complete [ 2423.267210] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2423.273088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2423.279091] Lustre: Skipped 1 previous similar message [ 2425.416385] LustreError: 141182:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2425.420986] LustreError: 141182:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2425.500317] Lustre: server umount lustre-MDT0000 complete [ 2427.560405] LustreError: 138585:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753366854 with bad export cookie 435645283375684514 [ 2427.562063] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2427.565805] LustreError: 138585:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2427.659518] LustreError: 141384:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2427.662725] LustreError: 141384:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2427.781964] Lustre: server umount lustre-MDT0001 complete [ 2434.453293] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 2436.194795] Key type lgssc unregistered [ 2436.460300] LNet: 141910:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2437.483525] LNet: Removed LNI 192.168.202.118@tcp [ 2437.984597] Key type .llcrypt unregistered [ 2437.986223] Key type ._llcrypt unregistered [ 2451.857063] Key type ._llcrypt registered [ 2451.858546] Key type .llcrypt registered [ 2451.922785] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 2452.381761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2452.412785] alg: No test for adler32 (adler32-zlib) [ 2453.358918] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 2453.505674] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 2453.509047] LNet: Accept secure, port 988 [ 2455.120103] Key type lgssc registered [ 2455.640325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2460.161541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2460.172262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2461.452085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2463.994663] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2469.906351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2470.209492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2472.829400] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2476.598387] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2478.970779] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2483.260142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2483.471610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2486.569411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 2487.102102] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2490.661298] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2495.789655] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 2511.593897] Lustre: lustre-OST0000: evicting bdfdea54-4d1f-4ee6-a6bb-86ea0ec5e4af (at 192.168.202.18@tcp) by administrative request [ 2511.599505] Lustre: Skipped 1 previous similar message [ 2537.952693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2537.956878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2537.963768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2542.049551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2542.052419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2542.056801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2542.067242] Lustre: Skipped 1 previous similar message [ 2547.168848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2551.264522] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2551.290161] LustreError: 146039:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2551.416212] Lustre: server umount lustre-OST0000 complete [ 2557.408643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2557.408675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2557.409329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2557.409332] Lustre: Skipped 1 previous similar message [ 2557.427640] Lustre: Skipped 1 previous similar message [ 2560.178620] LustreError: 146241:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2560.186127] LustreError: 146241:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2560.268090] Lustre: server umount lustre-MDT0000 complete [ 2562.528977] LustreError: 143272:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2562.539216] LustreError: 143272:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2562.850026] LustreError: 143252:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753366989 with bad export cookie 17584781384509305738 [ 2562.858060] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2562.858559] LustreError: 143252:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2562.952415] LustreError: 146443:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2562.955248] LustreError: 146443:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2563.083850] Lustre: server umount lustre-MDT0001 complete [ 2570.516795] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 2572.149386] Key type lgssc unregistered [ 2572.384395] LNet: 146969:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2573.413443] LNet: Removed LNI 192.168.202.118@tcp [ 2573.990491] Key type .llcrypt unregistered [ 2573.992226] Key type ._llcrypt unregistered [ 2613.237498] Key type ._llcrypt registered [ 2613.239197] Key type .llcrypt registered [ 2613.302677] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 2613.816872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2613.836247] alg: No test for adler32 (adler32-zlib) [ 2614.760752] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 2614.875648] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 2614.880156] LNet: Accept secure, port 988 [ 2616.487179] Key type lgssc registered [ 2617.177236] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2621.078904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2621.090542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2622.371148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2624.235359] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2628.731245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2628.936442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2631.137705] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2634.170977] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2636.251237] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2639.925632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2640.111074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2642.559523] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2644.929502] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2647.271272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2653.419502] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 2658.787264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2658.789289] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2658.798544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2658.804324] Lustre: Skipped 1 previous similar message [ 2660.275337] LustreError: 151656:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2660.405177] Lustre: server umount lustre-OST0000 complete [ 2663.905296] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2663.908122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2663.914798] Lustre: Skipped 2 previous similar messages [ 2668.439503] LustreError: 151857:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2668.443672] LustreError: 151857:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2668.495906] Lustre: server umount lustre-MDT0000 complete [ 2669.024316] LustreError: 151548:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2669.042052] LustreError: 151548:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2670.635954] LustreError: 149381:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753367097 with bad export cookie 13331389636202187289 [ 2670.640060] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2670.640787] LustreError: 149381:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2670.723546] LustreError: 152061:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2670.726380] LustreError: 152061:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2670.855893] Lustre: server umount lustre-MDT0001 complete [ 2677.256370] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 2678.713990] Key type lgssc unregistered [ 2678.875695] LNet: 152598:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2679.909579] LNet: Removed LNI 192.168.202.118@tcp [ 2680.340330] Key type .llcrypt unregistered [ 2680.342186] Key type ._llcrypt unregistered [ 2693.234178] Key type ._llcrypt registered [ 2693.236423] Key type .llcrypt registered [ 2693.321530] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 2693.823163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2693.831984] alg: No test for adler32 (adler32-zlib) [ 2694.741788] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 2694.856295] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 2694.859172] LNet: Accept secure, port 988 [ 2696.471168] Key type lgssc registered [ 2696.979262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2700.590676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2705.096573] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 10:25:31 (1753367131) [ 2713.233949] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 2717.478746] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 2721.977946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2724.791244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2727.313336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2730.055121] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2733.070100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2733.095963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2733.105648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2734.209695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2734.226613] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2734.267822] Lustre: lustre-MDT0000: new disk, initializing [ 2734.321299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2734.335063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2736.025369] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2741.128753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2741.163266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2741.214871] Lustre: 156766:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2741.251351] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2741.254124] Lustre: Skipped 1 previous similar message [ 2741.293058] Lustre: lustre-MDT0001: new disk, initializing [ 2741.334106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2741.353910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2741.358373] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2743.091529] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2746.912592] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2748.536724] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2751.626133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2751.658162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2751.784140] Lustre: lustre-OST0000: new disk, initializing [ 2751.787401] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2751.834947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2754.350028] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2757.157517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2757.167857] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2757.232549] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2758.389253] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2762.209784] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2762.210268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2762.213222] LustreError: Skipped 1 previous similar message [ 2762.219414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2762.224248] Lustre: Skipped 1 previous similar message [ 2764.140102] LustreError: 159330:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2764.226643] Lustre: server umount lustre-OST0000 complete [ 2766.819170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2766.825573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2766.831387] Lustre: Skipped 1 previous similar message [ 2766.833348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2768.002418] LustreError: 159536:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2768.004623] LustreError: 159536:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2768.037552] Lustre: server umount lustre-MDT0000 complete [ 2769.890414] LustreError: 156756:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753367196 with bad export cookie 9820614113508651303 [ 2769.894106] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2769.905812] LustreError: 156756:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2769.983605] LustreError: 159739:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2769.985962] LustreError: 159739:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2770.129474] Lustre: server umount lustre-MDT0001 complete [ 2775.595324] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 2776.896695] Key type lgssc unregistered [ 2777.049578] LNet: 160265:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2778.086606] LNet: Removed LNI 192.168.202.118@tcp [ 2778.532045] Key type .llcrypt unregistered [ 2778.533603] Key type ._llcrypt unregistered [ 2789.777780] Key type ._llcrypt registered [ 2789.779468] Key type .llcrypt registered [ 2789.839346] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 2790.366959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2790.386288] alg: No test for adler32 (adler32-zlib) [ 2791.270951] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 2791.370934] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 2791.373348] LNet: Accept secure, port 988 [ 2792.967188] Key type lgssc registered [ 2793.413949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2797.391775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2797.403768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2798.662158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2800.757569] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2805.989082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2806.190945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2808.421848] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2811.561133] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2813.674742] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2817.595895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2817.794452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2820.627364] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2823.558315] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2823.973870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2836.672118] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2836.686726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2836.694070] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2836.698277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2836.707480] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.118@tcp (at 0@lo) [ 2837.226266] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2837.242251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2837.250732] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2837.255507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2837.261977] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.118@tcp (at 0@lo) [ 2837.846548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2838.421773] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2838.425500] Lustre: Skipped 1 previous similar message [ 2838.436226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2838.444555] Lustre: Skipped 1 previous similar message [ 2838.447047] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2838.452266] Lustre: Skipped 1 previous similar message [ 2838.454497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2838.464563] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.118@tcp (at 0@lo) [ 2838.467442] Lustre: Skipped 1 previous similar message [ 2838.990523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2839.501323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2840.054846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2840.609988] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2840.613662] Lustre: Skipped 3 previous similar messages [ 2840.628987] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2840.636959] Lustre: Skipped 3 previous similar messages [ 2840.640369] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2840.647533] Lustre: Skipped 3 previous similar messages [ 2840.653817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2840.659505] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.118@tcp (at 0@lo) [ 2840.661913] Lustre: Skipped 3 previous similar messages [ 2841.239791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2841.824324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2842.377479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2842.902806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2843.442519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2843.957652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2844.489457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2844.974551] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2844.977854] Lustre: Skipped 7 previous similar messages [ 2844.990537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2844.996889] Lustre: Skipped 7 previous similar messages [ 2845.000857] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2845.003735] Lustre: Skipped 7 previous similar messages [ 2845.008055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2845.020250] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.118@tcp (at 0@lo) [ 2845.022837] Lustre: Skipped 7 previous similar messages [ 2845.524330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2846.049102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2846.638224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2847.203456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2847.754445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2848.310395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2848.890915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2849.403269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2849.980907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2850.533036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2851.048689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2851.588691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2852.081568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2852.580273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2853.099224] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2853.103240] Lustre: Skipped 14 previous similar messages [ 2853.112565] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2853.120500] Lustre: Skipped 14 previous similar messages [ 2853.124067] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2853.127151] Lustre: Skipped 14 previous similar messages [ 2853.129786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2853.137429] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.118@tcp (at 0@lo) [ 2853.140830] Lustre: Skipped 14 previous similar messages [ 2853.688085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2854.281748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2854.811847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2855.381844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2856.006664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2856.559650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2857.045932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2857.510233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2857.954718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2858.362906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2858.791709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2859.245506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2859.668664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2860.142185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2860.618955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2861.036964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2861.433659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2861.897289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2862.364848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2862.781134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2863.198541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2863.589433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2864.056880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2864.516066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2864.954953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2865.367465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2865.792464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2866.242369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2866.713752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2867.238509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2867.750568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2868.274595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2868.763374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2869.246808] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2869.251570] Lustre: Skipped 33 previous similar messages [ 2869.261636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2869.267599] Lustre: Skipped 33 previous similar messages [ 2869.270883] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2869.274218] Lustre: Skipped 33 previous similar messages [ 2869.277520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2869.283141] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.118@tcp (at 0@lo) [ 2869.287156] Lustre: Skipped 33 previous similar messages [ 2869.782518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2870.321652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2870.770805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2871.251632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2871.796418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2872.306725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2872.830447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2873.337094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2873.890449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2874.398414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2874.894694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2875.429412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2875.997868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2876.584527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2877.155975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2877.707622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2878.191649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2878.718040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2879.254967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2879.746703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2880.263767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2880.746980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2881.266552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2881.736111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2882.238846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2882.756705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2883.233776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2883.774280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2884.240238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2884.736108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2885.223302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2885.671582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2886.102350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2886.593856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2887.117847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2887.623052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2888.159710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2888.610818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2889.108460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2889.569856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2890.045174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2890.571481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2891.084824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2891.539774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2892.048027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2892.646914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2893.127238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2893.602315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2894.117929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2894.632183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2895.172956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2895.683477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2896.162209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2896.647328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2897.076906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2897.576877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2898.066394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2898.520328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2898.990971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2899.476365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2899.971864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2900.457481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2900.896890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2901.365691] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2901.369585] Lustre: Skipped 63 previous similar messages [ 2901.377667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2901.384550] Lustre: Skipped 63 previous similar messages [ 2901.388223] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2901.391436] Lustre: Skipped 63 previous similar messages [ 2901.393732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2901.399256] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.118@tcp (at 0@lo) [ 2901.403325] Lustre: Skipped 63 previous similar messages [ 2901.850745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2902.340348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2902.838061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2903.322356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2903.773496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2904.276429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2904.813174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2905.269845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2905.832103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2906.347988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2906.825507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2907.331767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2907.824338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2908.331519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2908.840802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2909.318700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2909.770800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2910.247454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2910.703993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2911.192106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2911.686823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2912.156978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2912.656165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2913.183625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2913.728878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2914.177556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2914.713072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2915.218216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2915.809694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2916.270396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2916.742310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2917.198787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2917.631887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2918.071713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2918.502113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2918.916155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2919.334695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2919.862764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2920.350739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2920.905716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2921.440651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2921.946685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2922.483823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2922.993561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2923.537586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2924.042884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2924.531365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2925.040531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2925.526707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2926.044645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2926.559035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2927.118943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2927.608076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2928.137750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2928.659327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2929.173370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2929.790177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2930.386757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2930.961072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2931.517417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2932.134592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2932.711415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2933.322066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2933.900651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2934.526371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2935.118645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2935.704812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2936.374425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2936.958455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2937.556974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2938.117326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2938.766062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2939.382226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2939.963452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2940.541346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2941.116923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2941.691975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2942.252396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2942.801747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2943.353486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2943.981826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2944.606287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2945.210640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2945.739637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2946.256380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2946.819306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2947.319181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2947.813575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2948.378758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2948.938517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2949.452066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2949.975527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2950.489292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2951.015349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2951.562413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2952.094421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2952.633868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2953.132941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2953.650695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2954.186319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2954.689865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2955.273864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2955.880711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2956.422161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2957.014874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2957.525731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2958.041953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2958.577387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2959.153477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2959.609890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2960.086168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2960.624845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2961.142152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2961.624362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2962.089130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2962.628408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2963.268483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2963.912525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2964.463654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2965.029301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2965.590558] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2965.592981] Lustre: Skipped 120 previous similar messages [ 2965.609753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2965.617438] Lustre: Skipped 120 previous similar messages [ 2965.619301] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2965.622632] Lustre: Skipped 120 previous similar messages [ 2965.625442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2965.632176] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.118@tcp (at 0@lo) [ 2965.636109] Lustre: Skipped 120 previous similar messages [ 2966.185024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2966.768851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2967.279941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2967.821818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2968.367417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2968.969057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2969.544113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2970.097264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2970.779338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2971.346773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2971.934441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2972.419448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2972.897805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2973.374311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2973.866242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2974.371727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2974.922399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2975.436811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2975.931858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2976.342731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2976.769179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2977.198800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2977.643034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2978.096146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2978.524802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2978.973852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2979.390603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2979.789658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2980.179744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2980.580731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2981.027634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2981.447241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2981.859062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2982.249080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2982.645854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2983.038638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2983.463911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2983.867621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2984.252935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2984.671448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2985.133124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2985.530049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2985.916951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2986.314178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2986.700993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2987.075579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2987.458128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2987.867499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2988.378255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2988.831738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2989.336132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2989.843773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2990.299255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2990.724588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2991.175182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2991.595612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2992.015013] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2992.444117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2992.851594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2993.239276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2993.723618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2994.164334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2994.562229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2994.972837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2995.396355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2995.859775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2996.334908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2996.771428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2997.202631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2997.629400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2998.074708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2998.488416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2998.896690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2999.310093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2999.726269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3000.161834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3000.599911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3001.028060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3001.542316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3002.091357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3002.605678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3003.102373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3003.563915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3004.047976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3004.504393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3004.992577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3005.451920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3005.887727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3006.374067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3006.825706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3007.293102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3007.712901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3008.173749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3008.642017] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3009.103790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3009.572144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3010.106543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3010.578692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3011.027398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3011.479917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3012.032454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3012.538521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3012.990343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3013.460619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3013.946504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3014.438694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3014.992055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3015.490571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3016.006118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3016.584176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3017.108839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3017.629567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3018.120696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3018.581348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3018.994338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3019.451942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3019.921176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3020.407481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3020.905943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3021.340519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3021.802496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3022.246319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3022.718061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3023.181029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3023.670448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3024.160101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3024.673569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3025.122876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3025.606157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3026.136328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3026.734951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3027.324738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3027.871619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3028.431871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3029.053978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3029.627923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3030.206810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3030.756484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3031.296199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3031.870826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3032.492506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3033.009715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3033.622177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3034.180512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3034.781381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3035.449835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3036.119106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3036.664504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3037.239941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3037.849044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3038.499815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3039.123801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3039.705630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3040.202908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3040.776676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3041.287067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3041.807092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3042.322982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3042.758533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3043.247213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3043.796278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3044.318962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3044.862778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3045.416267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3046.052826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3046.680591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3047.267318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3047.887218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3048.461841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3048.994745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3049.542589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3050.182065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3050.734524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3051.268264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3051.803448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3052.278724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3052.780395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3053.340830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3053.877953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3054.390053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3054.888653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3055.470403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3056.056388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3056.681882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3057.176639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3057.713155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3058.262593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3058.789241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3059.288943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3059.876523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3060.465326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3061.022794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3061.632869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3062.238149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3062.804545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3063.392604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3063.934783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3064.517144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3065.034636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3065.558281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3066.178052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3066.757979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3067.300762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3067.809095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3068.329834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3068.938352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3069.475690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3069.978046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3070.465145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3071.006465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3071.647145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3072.231499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3072.788835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3073.301675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3073.910286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3074.456798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3074.990462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3075.517666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3076.079856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3076.607303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3077.094808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3077.543956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3078.016702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3078.467977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3078.928150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3079.412025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3079.837385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3080.336959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3080.905991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3081.333030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3081.771073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3082.230965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3082.719406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3083.160332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3083.602715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3084.042838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3084.502389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3084.940827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3085.406144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3085.853183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3086.327664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3086.822484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3087.263259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3087.729055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3088.209148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3088.716314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3089.227498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3089.721802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3090.187303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3090.681072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3091.149516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3091.640189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3092.141065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3092.582114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3093.065146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3093.551244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3093.998520] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3094.001451] Lustre: Skipped 256 previous similar messages [ 3094.011846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3094.017414] Lustre: Skipped 256 previous similar messages [ 3094.020136] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3094.022513] Lustre: Skipped 256 previous similar messages [ 3094.024319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3094.029553] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.118@tcp (at 0@lo) [ 3094.032579] Lustre: Skipped 256 previous similar messages [ 3094.493628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3094.909957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3095.349336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3095.834181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3096.283776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3096.683568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3097.096122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3097.543172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3097.945476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3098.375611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3098.794614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3099.241851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3099.646316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3100.045988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3100.464385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3100.870895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3101.312850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3101.784936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3102.261585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3102.698649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3103.191672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3103.700115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3104.177152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3104.778458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3105.262714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3105.715387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3106.164606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3106.630499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3107.083737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3107.488199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3107.894216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3108.289183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3108.701337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3109.113375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3109.476431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3109.848549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3110.226844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3110.601738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3110.997535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3111.423874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3111.834333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3112.244991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3112.673165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3113.094237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3113.486799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3113.966261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3114.354789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3114.744977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3115.123575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3115.535351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3115.935122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3116.378978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3116.793093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3117.227448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3117.712565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3118.181341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3118.633796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3119.083405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3119.498979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3119.923027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3120.354203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3120.832033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3121.298270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3121.735304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3122.107688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3122.514805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3122.941313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3123.345567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3123.784068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3124.226143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3124.632403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3125.033536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3125.452862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3125.882241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3126.286020] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3126.702574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3127.111798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3127.529946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3128.018668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3128.423279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3128.836849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3129.259624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3129.654339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3130.088425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3130.524760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3130.951857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3131.374668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3131.834185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3132.249772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3132.713223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3133.206513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3133.653446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3134.146959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3134.593672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3135.017512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3135.464659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3135.923922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3136.393997] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3136.867828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3137.313524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3137.703666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3138.101465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3138.559752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3138.991624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3139.405797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3139.847615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3140.274541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3140.712381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3141.172387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3141.624293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3142.167413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3142.639508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3143.080796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3143.553902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3144.013196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3144.477328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3144.932639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3145.423399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3145.864770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3146.293995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3146.722249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3147.136712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3147.546524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3148.029932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3148.496794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3148.927253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3149.366732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3149.816917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3150.227296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3150.650719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3151.117969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3151.540207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3151.971719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3152.381684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3152.758782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3153.178880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3153.591935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3153.989502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3154.406263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3154.798345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3155.209625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3155.625467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3156.038245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3156.456471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3156.970733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3157.376832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3157.765171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3158.174888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3158.581084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3158.975041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3159.372244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3159.740724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3160.092741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3160.477384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3160.860551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3161.224394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3161.593093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3161.989788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3162.384647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3162.815630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3163.250226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3163.693978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3164.110103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3164.495604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3164.874113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3165.245492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3165.606460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3165.951906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3166.272878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3166.608263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3166.954190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3167.319977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3167.722891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3168.156314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3168.554319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3168.958772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3169.339856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3169.740101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3170.093900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3170.462906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3170.844792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3171.224348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3171.618703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3172.014429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3172.432134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3172.814077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3173.176931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3173.583831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3173.953722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3174.292939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3174.651328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3175.012464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3175.414951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3175.871202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3176.298028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3176.698972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3177.068883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3177.415416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3177.749237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3178.100755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3178.459879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3178.800795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3179.166666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3179.499893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3179.813065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3180.151489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3180.497851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3180.853739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3181.182187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3181.516769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3181.918076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3182.298429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3182.645758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3183.007953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3183.359408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3183.714066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3184.085447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3184.457382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3184.804761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3185.146768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3185.496721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3185.918689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3186.356522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3186.749724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3187.149638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3187.580202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3187.961305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3188.360664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3188.753766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3189.134154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3189.556069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3189.956251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3190.343423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3190.721282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3191.124337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3191.542157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3191.985741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3192.401554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3192.763860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3193.169595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3193.553990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3193.930438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3194.278729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3194.630204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3194.953023] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3195.276776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3195.616350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3195.962730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3196.299277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3196.718995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3197.081672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3197.421128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3197.763798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3198.169368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3198.585068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3198.965955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3199.342598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3199.721460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3200.087196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3200.451798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3200.798298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3201.131627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3201.446186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3201.797802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3202.132796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3202.459532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3202.835886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3203.199046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3203.560238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3203.925579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3204.265295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3204.607827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3204.981062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3205.323830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3205.683687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3206.030327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3206.381392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3206.745988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3207.107074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3207.459099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3207.809205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3208.131717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3208.477455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3208.879286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3209.250315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3209.633211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3209.986634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3210.368242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3210.704463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3211.047556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3211.381955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3211.726042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3212.125295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3212.489902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3212.830956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3213.186274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3213.528855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3213.892132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3214.234068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3214.579970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3214.904896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3215.225879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3215.554289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3215.913955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3216.234364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3216.566055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3216.890948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3217.259747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3217.597448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3217.938441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3218.274648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3218.606549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3218.961922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3219.307498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3219.654163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3219.998074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3220.342501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3220.687522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3221.021995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3221.351263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3221.735293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3222.085532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3222.420496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3222.763845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3223.117318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3223.448573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3223.807098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3224.152501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3224.515385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3224.854589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3225.218391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3225.556558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3225.920765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3226.265365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3226.629843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3226.965111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3227.293663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3227.607863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3227.936032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3228.247921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3228.576205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3228.915088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3229.293530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3229.630556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3229.990932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3230.326337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3230.669965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3231.009899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3231.331214] LustreError: 161647:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 3231.335127] LustreError: 161647:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 3231.336983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3231.668860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3231.981442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3232.300386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3232.624451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3232.955673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3233.295856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3233.676269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3234.017703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3234.372921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3234.736798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3235.072214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3235.428688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3235.777830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3236.135249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3236.474489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3236.834311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3237.182418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3237.561723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3237.946087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3238.304657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3238.636800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3238.987444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3239.331992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3239.684975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3240.044976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3240.385215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3240.728567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3241.059127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3241.382699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3241.739490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3242.079197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3242.463815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3242.821585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3243.181124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3243.540468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3243.874251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3244.232515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3244.588641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3244.944818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3245.287011] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3245.640643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3245.995497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3246.359335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3246.719746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3247.112656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3247.491233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3247.844703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3248.219135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3248.578525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3248.949621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3249.300533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3249.657897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3250.031188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3250.406604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3250.770368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3251.125563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3251.513632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3251.891287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3252.263160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3252.624991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3252.981410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3253.310480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3253.653704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3254.000821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3254.351254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3254.706751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3255.074482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3255.419915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3255.769066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3256.129935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3256.469973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3256.809616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3257.172081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3257.513848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3257.844683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3258.175127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3258.509058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3258.881375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3259.205371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3259.549581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3259.904673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3260.283310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3260.655404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3261.006950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3261.359561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3261.717501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3262.053127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3262.397221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3262.716903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3263.036203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3263.372323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3263.738521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3264.116833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3264.451635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3264.824136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3265.167660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3265.505819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3265.831014] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3266.188669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3266.556602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3266.914149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3267.281642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3267.634865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3267.982051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3268.345225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3268.714260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3269.073245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3269.433378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3269.800690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3270.149361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3270.500316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3270.854784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3271.190841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3271.521436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3271.889733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3272.239564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3272.576314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3272.913663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3273.269683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3273.641057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3273.994799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3274.333306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3274.664852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3274.990473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3275.337165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3275.678949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3276.033690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3276.382176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3276.723935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3277.085704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3277.448910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3277.780727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3278.093594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3278.428754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3278.768278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3279.147061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3279.518054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3279.874190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3280.248234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3280.630765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3281.009599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3281.430298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3281.866569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3282.331535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3553.889863] LustreError: 163215:0:(ofd_dev.c:1776:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 3553.894644] LustreError: 161647:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 3675.105902] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3675.107102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3675.111829] Lustre: Skipped 494 previous similar messages [ 3678.273110] LustreError: 215413:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3678.383331] Lustre: server umount lustre-OST0000 complete [ 3680.225414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3680.227713] Lustre: Skipped 2 previous similar messages [ 3685.343943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3685.345911] Lustre: Skipped 1 previous similar message [ 3686.087184] LustreError: 215614:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3686.089191] LustreError: 215614:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3686.180933] Lustre: server umount lustre-MDT0000 complete [ 3687.773383] LustreError: 161609:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753368114 with bad export cookie 16451662352889667305 [ 3687.778264] LustreError: 161609:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3687.778429] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3687.845219] LustreError: 215815:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3687.848264] LustreError: 215815:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3687.934460] Lustre: server umount lustre-MDT0001 complete [ 3693.138659] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 3694.434958] Key type lgssc unregistered [ 3694.602551] LNet: 216341:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3695.653333] LNet: Removed LNI 192.168.202.118@tcp [ 3695.995937] Key type .llcrypt unregistered [ 3695.997075] Key type ._llcrypt unregistered [ 3706.381589] Key type ._llcrypt registered [ 3706.382683] Key type .llcrypt registered [ 3706.422105] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 3706.815696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3706.846365] alg: No test for adler32 (adler32-zlib) [ 3707.717915] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 3707.808432] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 3707.810773] LNet: Accept secure, port 988 [ 3709.407204] Key type lgssc registered [ 3709.823096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3712.804317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3712.813623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3713.986160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3715.343764] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3718.483865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3718.624931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3720.061664] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3722.062703] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3723.418899] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3726.188470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3726.322160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3728.459642] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3730.422920] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3738.600566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49666 to 0x280000401:49697) [ 3744.028682] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3744.818977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3744.826674] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3744.829870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3744.833831] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.118@tcp (at 0@lo) [ 3744.833922] LustreError: 217722:0:(osp_precreate.c:1266:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 3746.927654] Lustre: lustre-OST0000: Not available for connect from 192.168.202.18@tcp (stopping) [ 3748.833495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3748.837595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3748.840220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3748.845825] Lustre: Skipped 1 previous similar message [ 3751.723926] LustreError: 220104:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3751.814983] Lustre: server umount lustre-OST0000 complete [ 3764.911114] LustreError: 217722:0:(osp_precreate.c:1266:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 3780.345197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3780.494144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3782.788695] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3784.918665] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3787.623940] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 3788.711024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3788.717715] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3788.721478] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.118@tcp (at 0@lo) [ 3788.739199] Lustre: Skipped 1 previous similar message [ 3789.094832] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c23496a6000.ost_server_uuid 50 [ 3789.688841] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9c23496a6000.ost_server_uuid in FULL state after 0 sec [ 3793.889369] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3793.890026] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3793.898327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3793.900632] Lustre: Skipped 2 previous similar messages [ 3796.844058] LustreError: 221416:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3796.846604] LustreError: 221416:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3796.909721] Lustre: server umount lustre-OST0000 complete [ 3799.008778] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3799.009273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3799.012766] Lustre: Skipped 2 previous similar messages [ 3804.128451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3804.130921] Lustre: Skipped 2 previous similar messages [ 3804.288417] LustreError: 221617:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3804.291208] LustreError: 221617:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3804.320134] Lustre: server umount lustre-MDT0000 complete [ 3805.762309] LustreError: 217684:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753368232 with bad export cookie 9316263097786815864 [ 3805.765472] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3805.767515] LustreError: 217684:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3805.934361] Lustre: server umount lustre-MDT0001 complete [ 3811.340748] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 3812.588578] Key type lgssc unregistered [ 3812.738503] LNet: 222344:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3813.797363] LNet: Removed LNI 192.168.202.118@tcp [ 3814.130638] Key type .llcrypt unregistered [ 3814.132153] Key type ._llcrypt unregistered [ 3827.643636] Key type ._llcrypt registered [ 3827.645511] Key type .llcrypt registered [ 3827.690861] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 3828.010601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3828.041547] alg: No test for adler32 (adler32-zlib) [ 3828.928649] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 3829.034155] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 3829.036460] LNet: Accept secure, port 988 [ 3830.631178] Key type lgssc registered [ 3831.082157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3834.954601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3834.962959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3836.152340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3837.613157] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3841.191344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3841.353156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3842.923846] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3844.194399] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3846.972906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3847.153410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3849.502783] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3853.075855] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3853.106815] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3853.177469] Lustre: lustre-OST0001: new disk, initializing [ 3853.180498] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3853.221865] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3855.636684] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3858.451676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3858.456619] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3858.490800] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3861.549588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49699 to 0x280000401:49729) [ 3861.909622] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3866.026968] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3881.953771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3881.957583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3881.959664] Lustre: Skipped 3 previous similar messages [ 3881.964427] Lustre: Skipped 3 previous similar messages [ 3887.073434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3887.076368] Lustre: Skipped 3 previous similar messages [ 3887.813441] LustreError: 227358:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3887.855695] Lustre: server umount lustre-MDT0000 complete [ 3889.442302] LustreError: 223998:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753368316 with bad export cookie 13890166652659211615 [ 3889.447614] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3889.447764] LustreError: 223998:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3889.526280] LustreError: 227561:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3889.528449] LustreError: 227561:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3889.633199] Lustre: server umount lustre-MDT0001 complete [ 3901.428077] LustreError: 227762:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3901.430567] LustreError: 227762:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3901.447853] Lustre: server umount lustre-OST0000 complete [ 3913.078522] LustreError: 227963:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3913.081587] LustreError: 227963:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3913.166465] Lustre: server umount lustre-OST0001 complete [ 3914.481287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3914.516845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3914.936418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3914.966989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3915.431548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3915.461359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3915.903366] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3915.927907] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3921.720725] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 3926.267238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3926.480874] LustreError: 229361:0:(ldlm_lib.c:1113:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3926.491597] LustreError: 229361:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3926.525385] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3928.151577] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3931.626443] LustreError: 229361:0:(ldlm_lib.c:1113:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3932.422779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3934.215820] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3935.557335] Lustre: 230458:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3938.688431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3938.876644] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3938.880103] Lustre: Skipped 1 previous similar message [ 3941.360070] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3942.949517] LustreError: 230815:0:(ldlm_lib.c:1113:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3945.453981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3947.869650] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3951.978677] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49732 to 0x280000401:49761) [ 3957.077992] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3958.544770] Lustre: 232294:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3968.480449] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3968.484297] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3968.490315] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3972.578941] Lustre: mylustre-MDT0000-lwp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3972.581028] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3972.584514] Lustre: Skipped 2 previous similar messages [ 3972.587220] Lustre: Skipped 2 previous similar messages [ 3974.594663] LustreError: 232601:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3974.596861] LustreError: 232601:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3974.632337] Lustre: server umount mylustre-MDT0000 complete [ 3976.193684] LustreError: 231622:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753368403 with bad export cookie 13890166652659214408 [ 3976.198991] LustreError: 231622:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3976.199994] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3976.379659] Lustre: server umount mylustre-MDT0001 complete [ 3987.188093] LustreError: 233003:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3987.190082] LustreError: 233003:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 3987.209762] Lustre: server umount mylustre-OST0000 complete [ 3999.043373] Lustre: server umount mylustre-OST0001 complete [ 4000.306692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4000.350472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4000.798858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4000.840863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4001.304928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4001.346747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4001.784074] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4001.823571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4007.493759] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 4011.588927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4011.792692] LustreError: 234603:0:(ldlm_lib.c:1113:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4011.822455] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4011.825389] Lustre: Skipped 1 previous similar message [ 4013.462324] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4017.124982] LustreError: 234603:0:(ldlm_lib.c:1113:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4017.250901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4019.217574] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4020.448062] Lustre: 235699:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4023.147369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4025.617352] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4027.427204] LustreError: 236632:0:(ldlm_lib.c:1113:target_handle_connect()) tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4029.042707] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4031.518486] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4036.210343] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49763 to 0x280000401:49793) [ 4040.365324] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4041.856809] Lustre: 237534:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4051.938862] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4051.941905] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 4051.946248] Lustre: Skipped 2 previous similar messages [ 4051.950398] Lustre: Skipped 3 previous similar messages [ 4055.683109] LustreError: 237781:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4055.686180] LustreError: 237781:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4055.725762] Lustre: server umount tfs-MDT0000 complete [ 4057.057518] LustreError: 234602:0:(ldlm_lib.c:1113:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4057.063555] LustreError: 234602:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4057.192197] LustreError: 235323:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753368484 with bad export cookie 13890166652659216074 [ 4057.196473] LustreError: 235323:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4057.197666] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4057.354253] Lustre: server umount tfs-MDT0001 complete [ 4068.362282] Lustre: server umount tfs-OST0000 complete [ 4079.307160] Lustre: server umount tfs-OST0001 complete [ 4080.626496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4080.664582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4081.117840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4081.154609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4081.578634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4081.604639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4082.037089] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4082.079620] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4087.732554] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 4092.026778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4092.247947] LustreError: 239782:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4092.290080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4092.293317] Lustre: Skipped 3 previous similar messages [ 4094.056778] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4097.761317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4099.556164] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4100.826117] Lustre: 240878:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4103.708576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4106.314807] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4110.200367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4112.837822] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4118.184894] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49763 to 0x280000401:49825) [ 4121.918150] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4123.442903] Lustre: 242711:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4131.085795] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 10:49:17 (1753368557) [ 4144.100305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4144.101031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4144.104304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4144.106144] Lustre: Skipped 2 previous similar messages [ 4149.124134] LustreError: 243078:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4149.127230] LustreError: 243078:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4149.164454] Lustre: server umount lustre-MDT0000 complete [ 4150.653731] LustreError: 239757:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753368577 with bad export cookie 13890166652659217516 [ 4150.658780] LustreError: 239757:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4150.659069] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4154.336698] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4154.340195] Lustre: Skipped 3 previous similar messages [ 4156.889438] Lustre: server umount lustre-MDT0001 complete [ 4164.478260] Lustre: server umount lustre-OST0000 complete [ 4172.223865] Lustre: server umount lustre-OST0001 complete [ 4174.657024] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 4178.230320] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 4182.090564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4184.569235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4186.796980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4189.246964] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4191.942841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4191.965585] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 4191.965585] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 4191.965585] [ 4191.979449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 4192.094361] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4192.109855] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4192.152265] Lustre: lustre-MDT0000: new disk, initializing [ 4192.187263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4192.189246] Lustre: Skipped 3 previous similar messages [ 4192.199645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4193.825613] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4198.839903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4198.865113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4198.903535] Lustre: 245850:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4198.931920] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4198.935251] Lustre: Skipped 1 previous similar message [ 4198.980587] Lustre: lustre-MDT0001: new disk, initializing [ 4199.043737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4199.047924] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4200.696785] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4205.515290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4205.539716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4205.665438] Lustre: lustre-OST0000: new disk, initializing [ 4205.668510] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4207.290309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4207.294202] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4207.336956] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4207.951971] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4212.346384] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4212.379469] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4212.449424] Lustre: lustre-OST0001: new disk, initializing [ 4212.452497] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4214.214916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4214.217687] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4214.250399] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4214.668945] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4219.069847] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 10:50:46 (1753368646) [ 4223.106671] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 10:50:49 (1753368649) [ 4234.725642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4234.733830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4234.737092] Lustre: Skipped 8 previous similar messages [ 4240.751823] LustreError: 249248:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4240.753742] LustreError: 249248:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4240.774217] Lustre: server umount lustre-OST0000 complete [ 4244.961754] LustreError: 249249:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4244.968033] LustreError: 249249:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 10 previous similar messages [ 4248.549123] Lustre: server umount lustre-MDT0000 complete [ 4250.092342] LustreError: 248243:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753368677 with bad export cookie 13890166652659218846 [ 4250.095848] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4250.096767] LustreError: 248243:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4256.349609] Lustre: server umount lustre-MDT0001 complete [ 4261.586798] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 4280.287216] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4280.384713] Lustre: server umount lustre-OST0001 complete [ 4282.622415] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 4289.487924] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 4293.204273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4295.716982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4297.737284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4300.093828] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4303.331666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4303.382855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4303.522867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4303.545577] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4303.593840] Lustre: lustre-MDT0000: new disk, initializing [ 4303.674705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4306.141562] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4315.272240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4315.346164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4315.468995] Lustre: 252685:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4315.940962] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4320.063695] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4328.057787] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4332.399634] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4339.266902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4339.337713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4339.646524] Lustre: lustre-OST0000: new disk, initializing [ 4339.653954] Lustre: Skipped 1 previous similar message [ 4339.660088] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4339.669286] Lustre: Skipped 2 previous similar messages [ 4339.767512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4339.774543] Lustre: Skipped 5 previous similar messages [ 4341.550218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4341.566019] Lustre: Skipped 1 previous similar message [ 4341.584990] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4341.715875] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4345.596338] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4352.926134] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4362.211073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4362.212794] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4362.219227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4362.224024] Lustre: Skipped 6 previous similar messages [ 4362.226770] Lustre: Skipped 6 previous similar messages [ 4364.737135] Lustre: server umount lustre-OST0000 complete [ 4367.328804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4367.332346] LustreError: Skipped 1 previous similar message [ 4373.197970] LustreError: 255308:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4373.201029] LustreError: 255308:0:(obd_class.h:479:obd_check_dev()) Skipped 29 previous similar messages [ 4373.254243] Lustre: server umount lustre-MDT0000 complete [ 4375.081334] LustreError: 252677:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753368802 with bad export cookie 13890166652659220316 [ 4375.085164] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4375.085872] LustreError: 252677:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4375.259367] Lustre: server umount lustre-MDT0001 complete [ 4382.048022] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 4383.495801] Key type lgssc unregistered [ 4383.657425] LNet: 256037:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4384.678308] LNet: Removed LNI 192.168.202.118@tcp [ 4385.074892] Key type .llcrypt unregistered [ 4385.076697] Key type ._llcrypt unregistered [ 4398.194150] Key type ._llcrypt registered [ 4398.196069] Key type .llcrypt registered [ 4398.268731] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 4398.666558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4398.674341] alg: No test for adler32 (adler32-zlib) [ 4399.605196] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 4399.738320] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 4399.741027] LNet: Accept secure, port 988 [ 4401.343423] Key type lgssc registered [ 4401.874610] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4406.145785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4406.167646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4407.472094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4409.366673] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4413.732354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4413.956042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4415.849706] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4418.963778] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4421.001968] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4425.005123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4425.222053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4428.073924] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4430.621238] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4431.060732] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 4431.966138] LustreError: 259732:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4432.051334] Lustre: server umount lustre-OST0000 complete [ 4434.400506] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4434.404234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4434.410888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4435.424667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4435.424707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4438.496562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4438.499160] Lustre: Skipped 2 previous similar messages [ 4440.009989] LustreError: 259932:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4440.012253] LustreError: 259932:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4440.072325] Lustre: server umount lustre-MDT0000 complete [ 4441.890362] LustreError: 257429:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753368868 with bad export cookie 8960478859797483332 [ 4441.893708] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4441.895908] LustreError: 257429:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4441.968674] LustreError: 260134:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4441.970627] LustreError: 260134:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4442.070767] Lustre: server umount lustre-MDT0001 complete [ 4447.090620] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 4448.590665] Key type lgssc unregistered [ 4448.780851] LNet: 260660:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4449.833761] LNet: Removed LNI 192.168.202.118@tcp [ 4450.271546] Key type .llcrypt unregistered [ 4450.272963] Key type ._llcrypt unregistered [ 4458.695619] Key type ._llcrypt registered [ 4458.697123] Key type .llcrypt registered [ 4458.758735] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 4459.163720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4459.224169] alg: No test for adler32 (adler32-zlib) [ 4460.137530] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 4460.267928] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 4460.273590] LNet: Accept secure, port 988 [ 4461.887205] Key type lgssc registered [ 4462.609250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4468.597964] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4468.615487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4485.152411] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4491.295732] LustreError: 261800:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.118@tcp: failed processing log, type 4: rc = -110 [ 4521.056175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4524.305267] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4527.599936] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4528.772110] LustreError: 262460:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 4528.880403] Lustre: server umount lustre-OST0000 complete [ 4537.707599] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 4539.730564] Key type lgssc unregistered [ 4539.997692] LNet: 263292:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4541.029429] LNet: Removed LNI 192.168.202.118@tcp [ 4541.510084] Key type .llcrypt unregistered [ 4541.512780] Key type ._llcrypt unregistered [ 4563.034670] Key type ._llcrypt registered [ 4563.037802] Key type .llcrypt registered [ 4563.122874] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 4563.670232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4563.686196] alg: No test for adler32 (adler32-zlib) [ 4564.650394] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 4564.819902] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 4564.824942] LNet: Accept secure, port 988 [ 4566.495182] Key type lgssc registered [ 4567.249841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4576.083954] loop0: detected capacity change from 0 to 204800000 [ 4576.198861] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4576.203644] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4576.410537] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4576.417329] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4576.451738] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4576.488074] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4576.518963] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4576.558837] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4576.608271] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4576.651523] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4577.351259] loop0: detected capacity change from 0 to 204800000 [ 4577.469935] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4577.862413] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4578.418415] loop0: detected capacity change from 0 to 204800000 [ 4578.652619] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4579.253307] loop0: detected capacity change from 0 to 204800000 [ 4579.472174] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4580.042786] loop0: detected capacity change from 0 to 204800000 [ 4580.066865] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 4582.430405] loop0: detected capacity change from 0 to 204800000 [ 4582.451693] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 4584.498176] loop0: detected capacity change from 0 to 204800000 [ 4584.520800] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 4586.785579] loop0: detected capacity change from 0 to 204800000 [ 4586.800764] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 4588.804808] loop0: detected capacity change from 0 to 204800000 [ 4588.850051] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4588.889946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4588.903426] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4588.941204] Lustre: 266660:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 4588.946834] Lustre: 266660:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 4588.952314] Lustre: 266660:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 4588.957708] Lustre: 266660:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 4588.966950] Lustre: 266660:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 4588.973109] Lustre: 266660:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 4588.979345] Lustre: 266660:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 4588.987180] Lustre: 266660:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 4588.993743] Lustre: 266660:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 4589.000269] Lustre: 266660:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 4589.006681] Lustre: 266660:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 4589.013400] Lustre: 266660:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 4589.021323] Lustre: 266660:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 4589.028129] Lustre: 266660:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 4592.907415] LustreError: 266927:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4592.980689] Lustre: server umount MGS complete [ 4593.477445] loop0: detected capacity change from 0 to 204800000 [ 4593.534865] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4593.801324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4593.839832] Lustre: lustre-MDT0000: Aborting client recovery [ 4594.336155] loop1: detected capacity change from 0 to 204800000 [ 4594.380684] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4594.443254] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4594.477173] Lustre: 267096:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 4594.483237] Lustre: 267096:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 4594.490925] Lustre: 267096:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 4594.498218] Lustre: 267096:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 4594.505205] Lustre: 267096:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 4594.510666] Lustre: 267096:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 4594.517246] Lustre: 267096:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 4594.530399] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4594.649171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4594.683373] Lustre: lustre-MDT0001: Aborting client recovery [ 4595.325099] loop2: detected capacity change from 0 to 204800000 [ 4595.416575] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4595.461435] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4595.487549] Lustre: 267180:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 4595.493444] Lustre: 267180:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 4595.498410] Lustre: 267180:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 4595.503576] Lustre: 267180:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 4595.509160] Lustre: 267180:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 4595.514573] Lustre: 267180:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 4595.519839] Lustre: 267180:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 4595.586578] Lustre: Found index 0 for lustre-OST0000, updating log [ 4595.655267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4595.664837] Lustre: lustre-OST0000: Aborting recovery [ 4596.211884] loop3: detected capacity change from 0 to 204800000 [ 4596.254978] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4596.288526] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4596.321442] Lustre: 267266:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 4596.329570] Lustre: 267266:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 4596.336363] Lustre: 267266:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 4596.342154] Lustre: 267266:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 4596.348140] Lustre: 267266:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 4596.353809] Lustre: 267266:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 4596.360357] Lustre: 267266:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 4596.442431] Lustre: lustre-OST0001: Aborting recovery [ 4601.847402] LustreError: 267039:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -108 [ 4603.878946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 4603.889302] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 4603.895770] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 4603.911865] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 4604.900522] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 4604.900726] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 4607.600816] LustreError: 267007:0:(mdt_identity.c:95:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 963us: rc = -2 [ 4608.112363] LustreError: 267004:0:(mdt_identity.c:95:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 536us: rc = -2 [ 4608.137393] LustreError: 267004:0:(mdt_identity.c:95:mdt_identity_do_upcall()) Skipped 35 previous similar messages [ 4620.263569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4620.265042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4620.265769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4620.268987] Lustre: Skipped 2 previous similar messages [ 4620.281441] Lustre: Skipped 3 previous similar messages [ 4625.376682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4625.388836] Lustre: Skipped 3 previous similar messages [ 4625.991128] LustreError: 267620:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4625.993294] LustreError: 267620:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4626.067038] Lustre: server umount lustre-MDT0000 complete [ 4626.678136] LustreError: 266990:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753369053 with bad export cookie 2751337827115078770 [ 4626.681968] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4626.691623] LustreError: 266990:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4630.505850] LustreError: 267003:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4630.512277] LustreError: 267003:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4630.516728] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4630.525949] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4630.528291] Lustre: Skipped 2 previous similar messages [ 4630.536433] Lustre: Skipped 1 previous similar message [ 4631.519528] LustreError: 267515:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4631.528760] LustreError: 267515:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4632.842301] LustreError: 267671:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 4632.844808] LustreError: 267671:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4632.997852] Lustre: server umount lustre-MDT0001 complete [ 4639.535280] LustreError: 267722:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4639.546051] LustreError: 267722:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4639.609736] Lustre: server umount lustre-OST0000 complete [ 4646.195144] LustreError: 267773:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4646.198581] LustreError: 267773:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4646.317464] Lustre: server umount lustre-OST0001 complete [ 4650.916795] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 10:57:57 (1753369077) [ 4666.548701] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 4673.125129] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 4680.329592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4684.745334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4688.373880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4692.602707] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4696.828848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4696.872882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4697.020694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4697.043236] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4697.095490] Lustre: lustre-MDT0000: new disk, initializing [ 4697.185324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4697.188633] Lustre: Skipped 1 previous similar message [ 4697.199141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4699.583355] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4706.798995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4706.836829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4706.894648] Lustre: 271304:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4706.942239] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4706.949413] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4706.950446] Lustre: Skipped 1 previous similar message [ 4706.955350] Lustre: Skipped 2 previous similar messages [ 4707.000184] Lustre: lustre-MDT0001: new disk, initializing [ 4707.069860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4707.098923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4707.109107] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4709.179490] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4713.722174] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4716.009847] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4720.246711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4720.285480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4720.456469] Lustre: lustre-OST0000: new disk, initializing [ 4720.468617] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4720.546895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4721.207985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4721.216611] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4721.268978] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4724.131064] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4728.621252] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4732.123193] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 4743.363806] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 4750.314608] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 4761.644717] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 4798.433073] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4798.433759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4798.438636] LustreError: Skipped 1 previous similar message [ 4798.441186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4798.448656] Lustre: Skipped 1 previous similar message [ 4798.455017] Lustre: Skipped 1 previous similar message [ 4808.166171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4808.168511] Lustre: Skipped 3 previous similar messages [ 4812.255318] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4812.275618] LustreError: 275020:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4812.278909] LustreError: 275020:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4812.387423] Lustre: server umount lustre-OST0000 complete [ 4814.817245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4814.821658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4820.436902] Lustre: server umount lustre-MDT0000 complete [ 4821.838693] LustreError: 273777:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753369248 with bad export cookie 2751337827115081479 [ 4821.842525] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4821.843153] LustreError: 273777:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4822.005105] Lustre: server umount lustre-MDT0001 complete [ 4825.768665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4827.427767] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4829.315691] LustreError: 276360:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4829.318836] LustreError: 276360:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4829.346113] Lustre: server umount MGS complete [ 4834.012716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4834.241260] LustreError: 276982:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4834.259533] LustreError: 276994:0:(qmt_pool.c:1513:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 4834.288155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4836.131302] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4839.393669] LustreError: 276982:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4839.998163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4841.865100] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4844.132704] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4845.685944] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4848.357124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4850.994238] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4853.275241] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4885.474806] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4885.475731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4885.481453] Lustre: Skipped 2 previous similar messages [ 4885.481789] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4885.483991] Lustre: Skipped 4 previous similar messages [ 4887.728408] LustreError: 279653:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4887.730216] LustreError: 279653:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4887.801029] Lustre: server umount lustre-OST0000 complete [ 4895.539651] Lustre: server umount lustre-MDT0000 complete [ 4895.712138] LustreError: 277701:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4895.719870] LustreError: 277701:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4897.353269] LustreError: 278601:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753369324 with bad export cookie 2751337827115083418 [ 4897.357086] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4897.358892] LustreError: 278601:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4897.583485] Lustre: server umount lustre-MDT0001 complete [ 4903.475816] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 4904.720939] Key type lgssc unregistered [ 4904.866822] LNet: 280582:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4905.895350] LNet: Removed LNI 192.168.202.118@tcp [ 4906.272670] Key type .llcrypt unregistered [ 4906.274221] Key type ._llcrypt unregistered [ 4921.348231] Key type ._llcrypt registered [ 4921.349627] Key type .llcrypt registered [ 4921.394370] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 4928.687767] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 4929.021656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4929.080076] alg: No test for adler32 (adler32-zlib) [ 4929.957529] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 4930.055296] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 4930.057452] LNet: Accept secure, port 988 [ 4931.655775] Key type lgssc registered [ 4932.146609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4935.930786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4938.899789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4941.537400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4944.258892] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4947.455467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4947.479771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4947.489357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4948.596938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4948.615346] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4948.655911] Lustre: lustre-MDT0000: new disk, initializing [ 4948.699327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4948.709207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4950.551368] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4955.894365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4955.927862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4955.977110] Lustre: 284468:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4956.004375] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4956.007019] Lustre: Skipped 1 previous similar message [ 4956.047468] Lustre: lustre-MDT0001: new disk, initializing [ 4956.099653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4956.127690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4956.132147] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4957.781972] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4961.518223] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4963.084540] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4965.938766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4965.969185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4966.092074] Lustre: lustre-OST0000: new disk, initializing [ 4966.095127] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4966.136203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4968.441718] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4972.046420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4972.050499] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4972.089018] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4972.155246] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4976.128172] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 4979.086152] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 4988.097575] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 4995.802855] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 5043.681787] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5043.682084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5043.684958] LustreError: Skipped 1 previous similar message [ 5043.691246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5043.694401] Lustre: Skipped 1 previous similar message [ 5044.974179] LustreError: 288020:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5045.048764] Lustre: server umount lustre-OST0000 complete [ 5048.290821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5048.293932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5048.299281] Lustre: Skipped 1 previous similar message [ 5048.302394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5053.920320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5053.923279] Lustre: Skipped 2 previous similar messages [ 5059.040346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5059.042987] Lustre: Skipped 1 previous similar message [ 5061.087315] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5061.131315] LustreError: 288221:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5061.134555] LustreError: 288221:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5061.172438] Lustre: server umount lustre-MDT0000 complete [ 5062.802507] LustreError: 284462:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753369489 with bad export cookie 7207121935924768605 [ 5062.805488] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5062.807920] LustreError: 284462:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5062.881034] LustreError: 288424:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5062.883142] LustreError: 288424:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5062.993695] Lustre: server umount lustre-MDT0001 complete [ 5066.838134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5068.483464] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5070.547696] LustreError: 289359:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5070.550381] LustreError: 289359:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5070.579126] Lustre: server umount MGS complete [ 5075.512600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5075.743476] LustreError: 289996:0:(qmt_pool.c:1513:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 5075.776370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5077.408857] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5080.888583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5081.050737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5082.646604] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5084.955568] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5086.470269] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5089.441893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5089.594937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5092.046711] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5094.408152] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5126.627491] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5126.629757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5126.638981] LustreError: Skipped 1 previous similar message [ 5126.639696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5126.647951] Lustre: Skipped 2 previous similar messages [ 5131.376760] LustreError: 292644:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5131.379829] LustreError: 292644:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5131.467217] Lustre: server umount lustre-OST0000 complete [ 5136.864993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5136.874680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5136.876651] Lustre: Skipped 1 previous similar message [ 5139.295208] Lustre: server umount lustre-MDT0000 complete [ 5140.748218] LustreError: 289965:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753369567 with bad export cookie 7207121935924770516 [ 5140.751058] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5140.753616] LustreError: 289965:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5140.830034] LustreError: 293047:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5140.832645] LustreError: 293047:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5140.935341] Lustre: server umount lustre-MDT0001 complete [ 5146.331283] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 5147.596789] Key type lgssc unregistered [ 5147.761567] LNet: 293573:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5148.773513] LNet: Removed LNI 192.168.202.118@tcp [ 5149.177564] Key type .llcrypt unregistered [ 5149.178854] Key type ._llcrypt unregistered [ 5157.667849] Key type ._llcrypt registered [ 5157.669053] Key type .llcrypt registered [ 5157.722892] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 5164.307280] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 5164.778242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5164.799737] alg: No test for adler32 (adler32-zlib) [ 5165.676492] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 5165.768537] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 5165.770652] LNet: Accept secure, port 988 [ 5167.359283] Key type lgssc registered [ 5167.756881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5171.449574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5174.054581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5177.689897] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 5181.190522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5183.775803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5183.798721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5183.807344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5184.918135] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5184.940191] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5184.984995] Lustre: lustre-MDT0000: new disk, initializing [ 5185.031480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5185.040820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5186.658903] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5191.626247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5191.654945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5191.759860] Lustre: lustre-OST0000: new disk, initializing [ 5191.762225] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5191.764617] Lustre: Skipped 1 previous similar message [ 5191.793665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5193.224089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 5193.228270] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 5193.241527] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 5194.009128] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5495.265220] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5495.270995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5497.348640] LustreError: 298463:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 5497.510268] Lustre: server umount lustre-MDT0000 complete [ 5498.888502] LustreError: 296709:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753369925 with bad export cookie 877931383316781799 [ 5498.891601] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5498.895075] LustreError: 296709:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5498.920067] LustreError: 298668:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5498.922354] LustreError: 298668:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5498.991145] Lustre: server umount lustre-OST0000 complete [ 5508.121209] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 5511.978718] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 5515.946711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5518.509480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5520.999251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5523.603289] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5526.110781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5526.132809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5526.212136] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5526.224576] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5526.264983] Lustre: lustre-MDT0000: new disk, initializing [ 5526.289482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5526.296665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5527.628997] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5532.071433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5532.099753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5532.142717] Lustre: 301690:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5532.172828] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5532.175342] Lustre: Skipped 1 previous similar message [ 5532.216296] Lustre: lustre-MDT0001: new disk, initializing [ 5532.262343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5532.277177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5532.281491] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5533.769193] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5537.257237] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5538.691884] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5541.408803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5541.437975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5541.540346] Lustre: lustre-OST0000: new disk, initializing [ 5541.542607] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5541.574474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5543.799826] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5543.946292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5543.951124] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5543.991928] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5547.195480] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5548.780244] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5548.879564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5550.293883] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5550.380232] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5554.148227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5554.149033] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5554.149238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5554.152901] Lustre: Skipped 1 previous similar message [ 5554.155684] LustreError: Skipped 1 previous similar message [ 5559.265742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5559.268291] Lustre: Skipped 2 previous similar messages [ 5564.385889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5564.388365] Lustre: Skipped 1 previous similar message [ 5565.407144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5565.424479] LustreError: 304597:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5565.426345] LustreError: 304597:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5565.494706] Lustre: server umount lustre-OST0000 complete [ 5567.968567] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5567.975648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5569.504240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5569.507064] Lustre: Skipped 2 previous similar messages [ 5579.744224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5579.746989] Lustre: Skipped 3 previous similar messages [ 5581.279132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5581.323569] LustreError: 304800:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5581.326512] LustreError: 304800:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5581.356396] Lustre: server umount lustre-MDT0000 complete [ 5582.748231] LustreError: 303424:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753370009 with bad export cookie 877931383319722576 [ 5582.750833] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5582.752868] LustreError: 303424:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5582.893713] Lustre: server umount lustre-MDT0001 complete [ 5584.143671] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [ 5584.717549] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 11:13:31 (1753370011) [ 5587.194279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5587.400043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5588.869515] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5592.063855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5593.722184] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5595.875930] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5597.211454] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5599.816811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5602.241759] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5604.434049] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5607.059048] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5607.086843] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5607.145097] Lustre: lustre-OST0001: new disk, initializing [ 5607.147995] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5607.184438] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5607.188830] Lustre: Skipped 2 previous similar messages [ 5609.239437] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5610.116392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5610.121438] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5610.138802] LustreError: 308500:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 5611.170533] LustreError: 308506:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 5611.174657] LustreError: 308506:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 5612.199306] LustreError: 308500:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 5612.204130] LustreError: 308500:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 5613.285889] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5613.371816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5614.242604] LustreError: 308506:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 5614.247558] LustreError: 308506:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 5614.793630] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c23496c7000.ost_server_uuid 50 [ 5615.368689] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9c23496c7000.ost_server_uuid in FULL state after 0 sec [ 5616.799198] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c23496c7000.ost_server_uuid 50 [ 5617.354669] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9c23496c7000.ost_server_uuid in FULL state after 0 sec [ 5618.347174] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5632.619907] Lustre: lustre-OST0001: Not available for connect from 192.168.202.18@tcp (stopping) [ 5632.997498] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5633.001971] Lustre: Skipped 2 previous similar messages [ 5635.377156] LustreError: 309354:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5635.386732] LustreError: 309354:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5635.414440] Lustre: server umount lustre-OST0001 complete [ 5638.114487] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5638.115182] LustreError: 307163:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5638.115309] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5638.120717] Lustre: Skipped 1 previous similar message [ 5638.127045] LustreError: 307163:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 5643.236451] LustreError: 307162:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5643.243269] LustreError: 307162:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5643.694801] LustreError: 309556:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5643.697181] LustreError: 309556:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5643.775576] Lustre: server umount lustre-OST0000 complete [ 5648.352776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5651.352109] Lustre: server umount lustre-MDT0000 complete [ 5652.763274] LustreError: 305556:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753370079 with bad export cookie 877931383319723444 [ 5652.766938] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5652.767715] LustreError: 305556:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5652.920070] Lustre: server umount lustre-MDT0001 complete [ 5658.187834] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 5659.416542] Key type lgssc unregistered [ 5659.580415] LNet: 310485:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5660.644901] LNet: Removed LNI 192.168.202.118@tcp [ 5660.983894] Key type .llcrypt unregistered [ 5660.985534] Key type ._llcrypt unregistered [ 5671.018434] Key type ._llcrypt registered [ 5671.020071] Key type .llcrypt registered [ 5671.062548] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 5671.498431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5671.527948] alg: No test for adler32 (adler32-zlib) [ 5672.394403] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 5672.483192] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 5672.485362] LNet: Accept secure, port 988 [ 5674.071146] Key type lgssc registered [ 5674.460482] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5677.452207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5677.465823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5678.686470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5680.227315] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5682.239713] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5684.925623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5685.094634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5686.623343] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5690.274214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5690.422530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5692.745499] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5694.500401] LustreError: 313393:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5694.507703] LustreError: 313393:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 5694.945396] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5697.600888] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5697.708056] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5700.048929] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5702.291173] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5704.175875] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5704.176680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 5710.582817] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5710.670554] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5790.175964] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5790.179046] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5790.184481] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5791.204397] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5791.204417] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5791.204765] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5794.412088] LustreError: 315240:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5794.428021] Lustre: server umount lustre-OST0001 complete [ 5796.322885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5796.324161] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5796.325614] LustreError: 313893:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5796.337157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5796.339569] Lustre: Skipped 2 previous similar messages [ 5801.443726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5801.443736] LustreError: 313392:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5801.446160] Lustre: Skipped 1 previous similar message [ 5801.452591] LustreError: 313392:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 5802.095193] LustreError: 315441:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 5802.097474] LustreError: 315441:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5802.160703] Lustre: server umount lustre-OST0000 complete [ 5806.560779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5806.561541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5806.564791] Lustre: Skipped 2 previous similar messages [ 5806.568666] Lustre: Skipped 1 previous similar message [ 5809.601259] LustreError: 315642:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5809.603862] LustreError: 315642:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5809.638411] Lustre: server umount lustre-MDT0000 complete [ 5810.863872] LustreError: 315141:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753370237 with bad export cookie 726992741728515174 [ 5810.867026] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5810.868883] LustreError: 315141:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5811.008159] Lustre: server umount lustre-MDT0001 complete [ 5815.629520] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 5816.713134] Key type lgssc unregistered [ 5816.853772] LNet: 316371:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5817.893300] LNet: Removed LNI 192.168.202.118@tcp [ 5818.200331] Key type .llcrypt unregistered [ 5818.201514] Key type ._llcrypt unregistered [ 5836.919832] Key type ._llcrypt registered [ 5836.921178] Key type .llcrypt registered [ 5836.966581] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 5837.345838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5837.428032] alg: No test for adler32 (adler32-zlib) [ 5838.291735] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 5838.386571] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 5838.388673] LNet: Accept secure, port 988 [ 5839.975160] Key type lgssc registered [ 5840.340063] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5843.596465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5843.604814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5844.772693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5846.192770] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5849.310635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5849.456653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5850.886817] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5851.934175] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5854.272027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5854.396234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5855.396031] LustreError: 320068:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5855.401221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 5855.402848] LustreError: 320068:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 5856.360537] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5859.438275] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5859.521144] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5861.179092] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 5861.626550] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5866.000136] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5869.721707] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5880.291251] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5880.292325] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5880.294786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5880.297462] Lustre: Skipped 3 previous similar messages [ 5880.299036] Lustre: Skipped 2 previous similar messages [ 5884.736037] LustreError: 322005:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5884.767067] Lustre: server umount lustre-MDT0000 complete [ 5885.409996] LustreError: 318619:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5885.417296] LustreError: 318619:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 5886.123238] LustreError: 321597:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753370313 with bad export cookie 11994465104199916990 [ 5886.126646] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5886.128628] LustreError: 321597:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5886.197252] LustreError: 322208:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 5886.199443] LustreError: 322208:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5886.284275] Lustre: server umount lustre-MDT0001 complete [ 5897.716077] LustreError: 322408:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 5897.718059] LustreError: 322408:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5897.736770] Lustre: server umount lustre-OST0000 complete [ 5909.427036] LustreError: 322610:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5909.429739] LustreError: 322610:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5909.510781] Lustre: server umount lustre-OST0001 complete [ 5915.581051] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 5919.186799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5919.329772] LustreError: 323776:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5919.357470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5920.685540] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5923.597143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5925.091268] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5926.082678] Lustre: 324874:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5928.246539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5928.377514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5928.380586] Lustre: Skipped 1 previous similar message [ 5929.379332] LustreError: 325228:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5929.382984] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5930.285551] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5933.194892] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5934.316046] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 5935.268139] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5938.392764] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5939.628422] Lustre: 326702:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5949.408067] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5949.411275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5949.416971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5951.807399] LustreError: 326808:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5951.810340] LustreError: 326808:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5951.843450] Lustre: server umount lustre-MDT0000 complete [ 5953.205356] LustreError: 324497:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753370380 with bad export cookie 11994465104199918649 [ 5953.209222] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5953.209913] LustreError: 324497:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5953.382564] Lustre: server umount lustre-MDT0001 complete [ 5964.216134] LustreError: 327210:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 5964.219043] LustreError: 327210:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 5964.238217] Lustre: server umount lustre-OST0000 complete [ 5975.804722] Lustre: server umount lustre-OST0001 complete [ 5977.068674] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [ 5977.618215] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [ 5978.248094] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 11:20:05 (1753370405) [ 5986.201829] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 5989.387320] loop0: detected capacity change from 0 to 1099511627776 [ 5989.558455] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 5989.561465] *** VALIDATE xfs *** [ 5989.565861] XFS (loop0): Mounting V5 Filesystem [ 5989.576159] XFS (loop0): Ending clean mount [ 5990.008896] loop1: detected capacity change from 0 to 18691697672192 [ 6005.509156] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6012.042208] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 6015.450143] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 6018.854495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6021.077105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6023.146496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6025.250739] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6027.603387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6027.622725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6027.698080] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6027.709332] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6027.747393] Lustre: lustre-MDT0000: new disk, initializing [ 6027.771728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6027.774046] Lustre: Skipped 1 previous similar message [ 6027.780426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6028.993133] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6032.867775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6032.890457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6032.925894] Lustre: 331753:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6032.949243] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6032.951457] Lustre: Skipped 1 previous similar message [ 6032.983105] Lustre: lustre-MDT0001: new disk, initializing [ 6033.024997] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6033.027431] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6034.190076] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6037.138962] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6038.251439] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6040.323220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6040.346144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6040.437621] Lustre: lustre-OST0000: new disk, initializing [ 6040.439141] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6042.102814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6042.105393] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6042.144678] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6042.279437] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6045.253917] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6046.689144] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6046.770667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6048.104098] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6048.175130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6050.859093] LustreError: 334658:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6050.861949] LustreError: 334658:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6050.921768] Lustre: server umount lustre-OST0000 complete [ 6052.321332] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6052.321855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6052.325205] Lustre: Skipped 4 previous similar messages [ 6052.327514] Lustre: Skipped 3 previous similar messages [ 6057.440665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6057.442389] Lustre: Skipped 2 previous similar messages [ 6062.561665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6062.563311] Lustre: Skipped 1 previous similar message [ 6062.590763] Lustre: server umount lustre-MDT0000 complete [ 6063.934210] LustreError: 331744:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753370490 with bad export cookie 11994465104199920203 [ 6063.937755] LustreError: 331744:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6063.937994] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6073.392780] Lustre: server umount lustre-MDT0001 complete [ 6076.090571] XFS (loop0): Unmounting Filesystem [ 6077.716419] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 11:21:44 (1753370504) [ 6080.200574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6080.340900] LustreError: 335877:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6080.364937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6080.367444] Lustre: Skipped 2 previous similar messages [ 6081.717321] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6084.755265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6086.331355] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6088.089391] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6089.293257] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6091.595280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6093.616858] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6095.511825] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6115.295954] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6115.299734] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6115.306129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6117.347234] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6120.683182] LustreError: 338224:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6120.686030] LustreError: 338224:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6120.753955] Lustre: server umount lustre-OST0000 complete [ 6122.464241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6122.469836] Lustre: Skipped 2 previous similar messages [ 6128.219390] Lustre: server umount lustre-MDT0000 complete [ 6129.507342] LustreError: 335857:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753370556 with bad export cookie 11994465104199921085 [ 6129.510331] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6129.511400] LustreError: 335857:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6129.652426] Lustre: server umount lustre-MDT0001 complete [ 6134.297132] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 6135.392062] Key type lgssc unregistered [ 6135.517387] LNet: 339154:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6136.553310] LNet: Removed LNI 192.168.202.118@tcp [ 6136.876377] Key type .llcrypt unregistered [ 6136.877565] Key type ._llcrypt unregistered [ 6147.590581] Key type ._llcrypt registered [ 6147.591894] Key type .llcrypt registered [ 6147.630563] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 6148.033937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6148.072805] alg: No test for adler32 (adler32-zlib) [ 6148.933574] Lustre: Lustre: Build Version: 2.16.57_3_g43b8dbf [ 6149.024565] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 6149.026817] LNet: Accept secure, port 988 [ 6150.623158] Key type lgssc registered [ 6151.002566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6153.784811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6153.794130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6154.967742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6156.271693] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6159.180729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6159.293565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6160.578280] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6162.370968] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6163.580957] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6165.906088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6166.036174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6168.120761] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6170.010718] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6184.418070] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6184.423930] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6184.426053] Lustre: Skipped 1 previous similar message [ 6185.581354] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.18@tcp (stopping) [ 6189.176982] LustreError: 342964:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6189.196759] Lustre: server umount lustre-MDT0001 complete [ 6189.539457] LustreError: 340665:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6189.546395] LustreError: 340665:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 6190.167041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6190.690705] LustreError: 340665:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6192.991786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6193.055406] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6193.057576] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6193.074529] Lustre: 340659:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6193.211651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6194.682052] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6198.244591] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 6201.318183] LustreError: 344089:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 6201.320669] LustreError: 344089:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 6201.323686] LustreError: 340662:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 6201.343386] LustreError: 344092:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.202.118@tcp: cfg command failed: rc = -17 [ 6201.347969] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 6201.347969] [ 6497.895027] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.18@tcp (stopping) [ 6500.320988] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6500.321756] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6500.328117] Lustre: Skipped 1 previous similar message [ 6500.330948] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6500.332655] Lustre: Skipped 1 previous similar message [ 6503.167800] LustreError: 355528:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6503.169809] LustreError: 355528:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6503.196881] Lustre: server umount lustre-MDT0001 complete [ 6504.182637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6505.440461] LustreError: 340670:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6505.444342] LustreError: 340670:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6506.721574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6506.751511] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6506.753376] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6506.766054] Lustre: 340659:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6506.879812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6508.146273] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6512.104532] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 6513.127459] LustreError: 356603:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 6513.130606] LustreError: 356603:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 6513.134108] LustreError: 340662:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 6513.135969] LustreError: 340662:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 6513.153853] LustreError: 356609:0:(obd_config.c:685:class_setup()) Device 8 already setup (type osp) [ 6513.156171] LustreError: 356609:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.202.118@tcp: cfg command failed: rc = -17 [ 6513.158971] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.202.118@tcp [ 6513.158971] [ 6814.178349] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6814.178616] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6814.180473] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6814.185573] Lustre: Skipped 2 previous similar messages [ 6814.187054] Lustre: Skipped 1 previous similar message [ 6816.638844] LustreError: 368139:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6816.641905] LustreError: 368139:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6816.666531] Lustre: server umount lustre-MDT0001 complete [ 6817.574249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6819.296427] LustreError: 340670:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6819.304197] LustreError: 340670:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6820.036121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6820.067241] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6820.068979] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6820.081118] Lustre: 340660:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6820.187947] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 6821.415165] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6825.444586] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 6829.542102] LustreError: 369358:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 6829.544997] LustreError: 369358:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 6829.547614] LustreError: 340662:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 6829.549424] LustreError: 340662:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 6829.568275] LustreError: 369361:0:(lod_lov.c:207:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 6829.575634] LustreError: 369361:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.202.118@tcp: cfg command failed: rc = -114 [ 6829.579974] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 6829.579974] [ 7137.762922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7137.763647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7137.767222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7137.770143] Lustre: Skipped 4 previous similar messages [ 7142.139609] LustreError: 380965:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7142.141080] LustreError: 380965:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7142.165648] Lustre: server umount lustre-MDT0000 complete [ 7142.881249] LustreError: 340665:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7142.887358] LustreError: 340665:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 7143.468382] LustreError: 343531:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753371570 with bad export cookie 14585728433303955534 [ 7143.471191] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7143.473018] LustreError: 343531:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7148.000307] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7148.003964] Lustre: Skipped 2 previous similar messages [ 7149.565475] LustreError: 381166:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7149.567620] LustreError: 381166:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7149.653036] Lustre: server umount lustre-MDT0001 complete [ 7151.064113] Lustre: server umount lustre-OST0000 complete [ 7154.272089] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 7157.376142] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 7160.646474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7162.911858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7164.892352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7167.045696] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7169.326121] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 11:39:56 (1753371596) [ 7171.427322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7171.449408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7171.533770] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7171.545690] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7171.582361] Lustre: lustre-MDT0000: new disk, initializing [ 7171.608907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7171.617341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7172.870162] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7176.816660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7176.837259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7176.866202] Lustre: 383881:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7176.886820] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7176.889184] Lustre: Skipped 1 previous similar message [ 7176.924498] Lustre: lustre-MDT0001: new disk, initializing [ 7176.954982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7176.967051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7176.970425] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7178.156182] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7181.156847] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7182.287744] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7184.426718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7184.449552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7184.547531] Lustre: lustre-OST0000: new disk, initializing [ 7184.549847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7184.578755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7186.352155] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7189.353972] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7194.105575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7194.109595] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7194.137916] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7197.663876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7197.665878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7197.671183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7197.673551] Lustre: Skipped 2 previous similar messages [ 7203.325156] LustreError: 386309:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7203.328080] LustreError: 386309:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 7203.355126] Lustre: server umount lustre-MDT0000 complete [ 7204.320034] LustreError: 383892:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7204.324722] LustreError: 383892:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 7204.531608] LustreError: 384657:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753371631 with bad export cookie 14585728433303957487 [ 7204.534473] LustreError: 384670:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [ 7204.535280] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7204.536093] LustreError: 384657:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7204.539410] LustreError: 384670:0:(client.c:1375:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003adf35d8 x1838542304411264/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 264/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7204.685476] Lustre: server umount lustre-MDT0001 complete [ 7215.221125] Lustre: server umount lustre-OST0000 complete [ 7224.798969] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 7228.130503] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 7231.781543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7234.252146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7236.087937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7238.149341] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7238.667551] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7073 sec ========== 11:41:05 (1753371665) [ 7239.242646] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:41:06 (1753371666) === [ 7240.334989] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:41:07 (1753371667) === [ 7253.816953] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 7254.798938] Key type lgssc unregistered [ 7254.922300] LNet: 391789:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7255.973312] LNet: Removed LNI 192.168.202.118@tcp [ 7256.250208] Key type .llcrypt unregistered [ 7256.251520] Key type ._llcrypt unregistered