[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 495795977 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002343] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.011009] LSM: Security Framework initializing [ 0.012048] Yama: becoming mindful. [ 0.013039] SELinux: Initializing. [ 0.014075] *** VALIDATE selinux *** [ 0.022419] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026711] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027158] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029112] *** VALIDATE tmpfs *** [ 0.030458] *** VALIDATE proc *** [ 0.032236] *** VALIDATE cgroup *** [ 0.033014] *** VALIDATE cgroup2 *** [ 0.034296] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035162] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037029] Spectre V2 : User space: Vulnerable [ 0.038014] Speculative Store Bypass: Vulnerable [ 0.041006] debug: unmapping init [mem 0xffffffffa3459000-0xffffffffa3460fff] [ 0.043175] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044721] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045021] ... version: 2 [ 0.046011] ... bit width: 48 [ 0.047015] ... generic registers: 4 [ 0.048012] ... value mask: 0000ffffffffffff [ 0.049014] ... max period: 00007fffffffffff [ 0.050014] ... fixed-purpose events: 3 [ 0.051014] ... event mask: 000000070000000f [ 0.052311] rcu: Hierarchical SRCU implementation. [ 0.054434] smp: Bringing up secondary CPUs ... [ 0.055636] x86: Booting SMP configuration: [ 0.056028] .... node #0, CPUs: #1 #2 #3 [ 0.059516] smp: Brought up 1 node, 4 CPUs [ 0.061013] smpboot: Max logical packages: 1 [ 0.062035] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.146814] node 0 deferred pages initialised in 82ms [ 0.150105] devtmpfs: initialized [ 0.151316] x86/mm: Memory block size: 128MB [ 0.154862] gcov: version magic: 0x41383552 [ 0.157176] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.161110] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.163311] pinctrl core: initialized pinctrl subsystem [ 0.165177] [ 0.165675] ************************************************************* [ 0.168015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170011] ** ** [ 0.172012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.174012] ** ** [ 0.176013] ** This means that this kernel is built to expose internal ** [ 0.179014] ** IOMMU data structures, which may compromise security on ** [ 0.181011] ** your system. ** [ 0.183014] ** ** [ 0.185013] ** If you see this message and you are not debugging the ** [ 0.187014] ** kernel, report this immediately to your vendor! ** [ 0.190014] ** ** [ 0.192012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.194013] ************************************************************* [ 0.196651] NET: Registered protocol family 16 [ 0.198399] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.200062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.203080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.206106] cpuidle: using governor menu [ 0.207818] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.208458] PCI: Using configuration type 1 for base access [ 0.209132] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.217057] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.219029] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.223112] cryptd: max_cpu_qlen set to 1000 [ 0.227309] ACPI: Added _OSI(Module Device) [ 0.231018] ACPI: Added _OSI(Processor Device) [ 0.232012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.234012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.239244] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.246242] ACPI: Interpreter enabled [ 0.247148] ACPI: PM: (supports S0 S3 S4 S5) [ 0.249015] ACPI: Using IOAPIC for interrupt routing [ 0.251125] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.255473] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.266589] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.269046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.272034] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.275085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.280333] acpiphp: Slot [2] registered [ 0.282120] acpiphp: Slot [5] registered [ 0.283120] acpiphp: Slot [6] registered [ 0.284143] acpiphp: Slot [7] registered [ 0.286152] acpiphp: Slot [8] registered [ 0.287115] acpiphp: Slot [9] registered [ 0.289103] acpiphp: Slot [10] registered [ 0.290101] acpiphp: Slot [3] registered [ 0.292093] acpiphp: Slot [4] registered [ 0.293084] acpiphp: Slot [11] registered [ 0.294119] acpiphp: Slot [12] registered [ 0.296098] acpiphp: Slot [13] registered [ 0.297094] acpiphp: Slot [14] registered [ 0.299118] acpiphp: Slot [15] registered [ 0.300094] acpiphp: Slot [16] registered [ 0.302124] acpiphp: Slot [17] registered [ 0.303107] acpiphp: Slot [18] registered [ 0.305103] acpiphp: Slot [19] registered [ 0.306094] acpiphp: Slot [20] registered [ 0.307093] acpiphp: Slot [21] registered [ 0.308079] acpiphp: Slot [22] registered [ 0.309081] acpiphp: Slot [23] registered [ 0.311123] acpiphp: Slot [24] registered [ 0.312100] acpiphp: Slot [25] registered [ 0.314084] acpiphp: Slot [26] registered [ 0.315133] acpiphp: Slot [27] registered [ 0.317115] acpiphp: Slot [28] registered [ 0.318102] acpiphp: Slot [29] registered [ 0.320095] acpiphp: Slot [30] registered [ 0.321114] acpiphp: Slot [31] registered [ 0.323079] PCI host bridge to bus 0000:00 [ 0.325020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.327024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.329022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.332028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.335023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.337020] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.339168] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.341896] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.345409] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.355825] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.360534] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.363025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.365016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.368019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.370577] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.373836] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.376053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.379617] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.385016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.397014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.403016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.409908] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.418017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.424018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.448017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.457964] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.471017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.482016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.510019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.523000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.530016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.537017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.602019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.611729] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.620023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.627019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.640016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.648223] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.653014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.662017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.682014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.691165] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.697018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.708030] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.728022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.742293] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.744323] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.747440] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.750385] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.752237] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.756090] iommu: Default domain type: Passthrough [ 0.757391] SCSI subsystem initialized [ 0.759152] ACPI: bus type USB registered [ 0.761136] usbcore: registered new interface driver usbfs [ 0.763088] usbcore: registered new interface driver hub [ 0.765086] usbcore: registered new device driver usb [ 0.767170] pps_core: LinuxPPS API ver. 1 registered [ 0.768010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.771062] PTP clock support registered [ 0.773132] EDAC MC: Ver: 3.0.0 [ 0.776133] PCI: Using ACPI for IRQ routing [ 0.778848] NetLabel: Initializing [ 0.780013] NetLabel: domain hash size = 128 [ 0.781010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.783085] NetLabel: unlabeled traffic allowed by default [ 0.786087] vgaarb: loaded [ 0.787270] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.789013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.797127] clocksource: Switched to clocksource kvm-clock [ 0.902537] VFS: Disk quotas dquot_6.6.0 [ 0.903621] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.906094] *** VALIDATE ramfs *** [ 0.907391] *** VALIDATE hugetlbfs *** [ 0.909226] pnp: PnP ACPI init [ 0.911559] pnp: PnP ACPI: found 6 devices [ 0.927448] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.930405] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.932331] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.934411] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.936617] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.938843] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.941731] NET: Registered protocol family 2 [ 0.944162] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.949285] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.953608] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.958897] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.962577] TCP: Hash tables configured (established 65536 bind 65536) [ 0.965529] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.968208] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.970644] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.972785] NET: Registered protocol family 1 [ 0.974854] RPC: Registered named UNIX socket transport module. [ 0.976419] RPC: Registered udp transport module. [ 0.977557] RPC: Registered tcp transport module. [ 0.978965] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.981159] NET: Registered protocol family 44 [ 0.982976] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.985384] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.987429] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.989727] PCI: CLS 0 bytes, default 64 [ 0.992059] Unpacking initramfs... [ 2.361279] debug: unmapping init [mem 0xffff97ad7cc54000-0xffff97ad7ffbffff] [ 2.364838] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.367038] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.369661] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.848059] Initialise system trusted keyrings [ 2.849825] Key type blacklist registered [ 2.851593] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.860444] zbud: loaded [ 2.863151] *** VALIDATE nfs *** [ 2.864258] *** VALIDATE nfs4 *** [ 2.865673] pstore: using deflate compression [ 2.869053] Platform Keyring initialized [ 2.963729] NET: Registered protocol family 38 [ 2.965348] Key type asymmetric registered [ 2.966548] Asymmetric key parser 'x509' registered [ 2.968029] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.971329] io scheduler mq-deadline registered [ 2.972888] io scheduler kyber registered [ 2.974388] io scheduler bfq registered [ 2.976205] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.978810] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.981669] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.984508] ACPI: Power Button [PWRF] [ 2.989925] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.999050] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.029176] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.036993] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.069923] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.100426] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.129488] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.134754] Non-volatile memory driver v1.3 [ 3.136368] Linux agpgart interface v0.103 [ 3.170513] virtio_blk virtio1: [vda] 150616 512-byte logical blocks (77.1 MB/73.5 MiB) [ 3.174066] vda: detected capacity change from 0 to 77115392 [ 3.189274] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.191797] vdb: detected capacity change from 0 to 1073741824 [ 3.207544] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.210339] vdc: detected capacity change from 0 to 2621440000 [ 3.221475] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.223694] vdd: detected capacity change from 0 to 2621440000 [ 3.237096] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.240775] vde: detected capacity change from 0 to 4294967296 [ 3.260936] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.264070] vdf: detected capacity change from 0 to 4294967296 [ 3.273538] libphy: Fixed MDIO Bus: probed [ 3.280399] usbcore: registered new interface driver usbserial_generic [ 3.283192] usbserial: USB Serial support registered for generic [ 3.285901] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.290557] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.292613] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.294742] mousedev: PS/2 mouse device common for all mice [ 3.297623] rtc_cmos 00:05: RTC can wake from S4 [ 3.300216] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.305715] rtc_cmos 00:05: registered as rtc0 [ 3.307669] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.308153] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.310710] intel_pstate: CPU model not supported [ 3.315565] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.320413] hid: raw HID events driver (C) Jiri Kosina [ 3.322646] usbcore: registered new interface driver usbhid [ 3.324617] usbhid: USB HID core driver [ 3.326170] drop_monitor: Initializing network drop monitor service [ 3.328357] Initializing XFRM netlink socket [ 3.330108] NET: Registered protocol family 10 [ 3.332702] Segment Routing with IPv6 [ 3.334134] NET: Registered protocol family 17 [ 3.336608] mpls_gso: MPLS GSO support [ 3.341710] RAS: Correctable Errors collector initialized. [ 3.344110] AVX version of gcm_enc/dec engaged. [ 3.345698] AES CTR mode by8 optimization enabled [ 3.410889] sched_clock: Marking stable (3410866998, 0)->(4285258646, -874391648) [ 3.414156] registered taskstats version 1 [ 3.416121] Loading compiled-in X.509 certificates [ 3.418520] zswap: loaded using pool lzo/zbud [ 3.439054] Key type big_key registered [ 3.451882] Key type encrypted registered [ 3.453617] ima: No TPM chip found, activating TPM-bypass! [ 3.455509] ima: Allocated hash algorithm: sha1 [ 3.456929] ima: No architecture policies found [ 3.458668] evm: Initialising EVM extended attributes: [ 3.460550] evm: security.selinux [ 3.461751] evm: security.ima [ 3.462834] evm: security.capability [ 3.464073] evm: HMAC attrs: 0x1 [ 3.466371] rtc_cmos 00:05: setting system clock to 2026-10-02 19:51:51 UTC (1790970711) [ 3.472715] debug: unmapping init [mem 0xffffffffa4403000-0xffffffffa45fffff] [ 3.475735] debug: unmapping init [mem 0xffffffffa3182000-0xffffffffa3458fff] [ 3.484780] Write protecting the kernel read-only data: 28672k [ 3.488434] debug: unmapping init [mem 0xffffffffa1803000-0xffffffffa19fffff] [ 3.491331] debug: unmapping init [mem 0xffffffffa2114000-0xffffffffa21fffff] [ 3.528990] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.537531] systemd[1]: Detected virtualization kvm. [ 3.538943] systemd[1]: Detected architecture x86-64. [ 3.540258] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.568522] systemd[1]: No hostname configured. [ 3.570063] systemd[1]: Set hostname to . [ 3.571957] random: systemd: uninitialized urandom read (16 bytes read) [ 3.574218] systemd[1]: Initializing machine ID from random generator. [ 3.628533] random: ln: uninitialized urandom read (6 bytes read) [ 3.715317] random: systemd: uninitialized urandom read (16 bytes read) [ 3.719403] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.724590] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.729727] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.373259] device-mapper: uevent: version 1.0.3 [ 4.375570] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.128306] virtio_net virtio0 ens2: renamed from eth0 [ 5.158952] random: fast init done [ 5.183718] scsi host0: ata_piix [ 5.204811] scsi host1: ata_piix [ 5.206612] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.209118] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.824278] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.970638] random: crng init done [ 9.972179] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.410278] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.598288] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.851240] SELinux: Disabled at runtime. [ 11.913231] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.920811] systemd[1]: Detected virtualization kvm. [ 11.922284] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.453127] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.456828] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.462547] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.468503] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.472543] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.481207] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.493304] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK [0[ 12.546757] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS m] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.948782] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.270471] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.338738] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.506947] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.528845] EDAC sbridge: Ver: 1.1.2 [ 15.206608] Key type dns_resolver registered [ 15.512786] NFS: Registering the id_resolver key type [ 15.515230] Key type id_resolver registered [ 15.517187] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg220-server login: [ 74.061831] libcfs: loading out-of-tree module taints kernel. [ 74.118815] Key type ._llcrypt registered [ 74.120953] Key type .llcrypt registered [ 74.231967] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 92.076701] hrtimer: interrupt took 17686982 ns [ 95.795823] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 97.615343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 97.627746] alg: No test for adler32 (adler32-zlib) [ 99.156235] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 100.284566] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 102.032169] Key type lgssc registered [ 104.476760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 127.165812] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 181.510199] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 194.792577] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 194.823136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 196.307223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 196.353987] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 196.422386] Lustre: lustre-MDT0000: new disk, initializing [ 196.498492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.513871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 201.713249] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.061228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 217.180915] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 217.265977] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 217.272459] Lustre: Skipped 1 previous similar message [ 217.394981] Lustre: lustre-MDT0001: new disk, initializing [ 217.503810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 217.561996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 217.570285] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 222.280141] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 227.650431] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 236.777892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 237.149056] Lustre: lustre-OST0000: new disk, initializing [ 237.153887] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 237.162823] Lustre: 8455:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 237.266038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 239.558959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 239.564924] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 239.677771] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 244.481385] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 259.762304] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 259.899784] Lustre: lustre-OST0001: new disk, initializing [ 259.902435] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 259.910328] Lustre: 9528:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 259.976655] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 265.903707] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 267.289776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 267.310692] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 267.381040] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 277.111695] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 281.299047] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 288.951731] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing check_logdir /tmp/testlogs/ [ 295.724189] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing yml_node [ 301.971635] Lustre: DEBUG MARKER: Client: 2.17.59.2 [ 305.124264] Lustre: DEBUG MARKER: MDS: 2.17.59.2 [ 308.457520] Lustre: DEBUG MARKER: OSS: 2.17.59.2 [ 310.906526] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 2 15:56:57 EDT 2026 [ 331.642759] Lustre: DEBUG MARKER: only running tests: 76-120 [ 333.463784] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 335.605934] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 354.275639] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 354.277581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 354.303781] Lustre: Skipped 1 previous similar message [ 354.321069] Lustre: Skipped 3 previous similar messages [ 358.658514] Lustre: server umount lustre-MDT0000 complete [ 363.710799] LustreError: 7465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790971071 with bad export cookie 2245389913144573978 [ 363.719103] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 363.727814] LustreError: 7465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 364.511955] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 364.512703] LustreError: 6523:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 364.512869] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 364.550670] Lustre: Skipped 3 previous similar messages [ 364.595396] LustreError: 6523:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 368.544950] LustreError: 6524:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 368.545362] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 368.590482] Lustre: Skipped 2 previous similar messages [ 373.675141] LustreError: 6524:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 373.675329] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 373.689026] LustreError: 6524:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 378.337472] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 378.758611] Lustre: server umount lustre-MDT0001 complete [ 383.226111] Lustre: server umount lustre-OST0000 complete [ 387.812907] Lustre: server umount lustre-OST0001 complete [ 395.579155] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 405.927761] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 446.839753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 447.230216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 447.293729] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 447.400323] Lustre: lustre-MDT0000: new disk, initializing [ 447.502576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 447.526731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 452.505762] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 462.059249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 462.169638] Lustre: 15156:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 462.230270] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 462.233652] Lustre: Skipped 1 previous similar message [ 462.327506] Lustre: lustre-MDT0001: new disk, initializing [ 462.435381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 462.490917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 462.505314] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 467.038691] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 477.579386] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 483.926839] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 492.355263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 492.748953] Lustre: lustre-OST0000: new disk, initializing [ 492.751442] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 492.755621] Lustre: 17126:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 492.900908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 494.257458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 494.268692] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 494.423485] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 499.429520] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 511.462946] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 517.498213] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 517.820559] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 523.909232] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 524.150823] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 530.405146] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 530.408715] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.427414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 530.427422] Lustre: Skipped 1 previous similar message [ 530.447064] Lustre: Skipped 1 previous similar message [ 540.128526] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 540.550942] Lustre: server umount lustre-OST0000 complete [ 544.224656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 544.230848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 544.244845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 544.253714] Lustre: Skipped 3 previous similar messages [ 550.432314] Lustre: server umount lustre-MDT0000 complete [ 550.880869] LustreError: 15163:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.911746] LustreError: 15163:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 555.261420] LustreError: 15148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790971263 with bad export cookie 2245389913144575511 [ 555.269873] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 555.272012] LustreError: 15148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 555.715548] Lustre: server umount lustre-MDT0001 complete [ 562.686019] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 16:01:09 (1790971269) [ 572.287896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 573.025082] LustreError: 19406:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.125807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 578.673248] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 583.649098] LustreError: 19407:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 583.677461] LustreError: 19407:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 586.748685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 591.670739] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 599.591457] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 605.055040] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 611.502823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 619.856537] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 628.416336] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 632.216062] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 638.945240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 638.956651] Lustre: Skipped 1 previous similar message [ 638.971424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 638.977431] Lustre: Skipped 3 previous similar messages [ 643.554510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 645.163890] Lustre: server umount lustre-MDT0000 complete [ 648.579236] LustreError: 21247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790971356 with bad export cookie 2245389913144576393 [ 648.583348] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 648.587952] LustreError: 21247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 648.905884] Lustre: server umount lustre-MDT0001 complete [ 662.316219] Lustre: server umount lustre-OST0000 complete [ 685.620704] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 696.106335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 696.343773] LustreError: 23791:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 696.375312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 696.378811] Lustre: Skipped 2 previous similar messages [ 702.398410] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 712.078054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 718.144618] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 721.687034] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 728.249649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 734.625521] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 742.913719] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 743.191615] Lustre: lustre-OST0001: new disk, initializing [ 743.195266] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 743.202231] Lustre: 26108:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 745.227594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 745.236930] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 745.259993] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 749.014504] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 758.126309] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 761.826399] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 774.115308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 774.135471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 774.151898] Lustre: Skipped 2 previous similar messages [ 774.162149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 774.165340] Lustre: Skipped 4 previous similar messages [ 780.225822] Lustre: server umount lustre-MDT0000 complete [ 781.290813] LustreError: 24944:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 781.312732] LustreError: 24944:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 783.834823] LustreError: 23772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790971491 with bad export cookie 2245389913144577233 [ 783.851323] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 784.204222] Lustre: server umount lustre-MDT0001 complete [ 798.683982] Lustre: server umount lustre-OST0000 complete [ 812.915487] Lustre: server umount lustre-OST0001 complete [ 828.725974] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 838.267490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 838.709258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 838.716511] Lustre: Skipped 3 previous similar messages [ 843.279153] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 848.870047] LustreError: 29129:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 848.883016] LustreError: 29129:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 851.667195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 857.333156] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 860.550773] Lustre: 30276:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 867.537894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 874.989267] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 884.610918] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 891.954745] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 899.574862] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 903.584550] Lustre: 32159:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 913.379794] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 913.403298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 913.421169] Lustre: Skipped 3 previous similar messages [ 913.429375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 913.444795] Lustre: Skipped 4 previous similar messages [ 918.258922] Lustre: server umount lustre-MDT0000 complete [ 923.150576] LustreError: 29111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790971631 with bad export cookie 2245389913144578528 [ 923.173582] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 923.184328] LustreError: 29111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 923.812888] Lustre: server umount lustre-MDT0001 complete [ 938.501270] Lustre: server umount lustre-OST0000 complete [ 952.709212] Lustre: server umount lustre-OST0001 complete [ 961.841459] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 16:07:48 (1790971668) [ 989.758027] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1000.605677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1001.064347] LustreError: 34963:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1001.085340] LustreError: 34963:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 1005.610451] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1014.089300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1021.283066] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1025.184893] Lustre: 36112:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1035.027643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1042.179215] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1052.234445] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1058.896855] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1065.833995] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1069.646265] Lustre: 37991:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1076.195423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1076.209222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1076.227836] Lustre: Skipped 3 previous similar messages [ 1076.238613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1076.251085] Lustre: Skipped 4 previous similar messages [ 1081.501452] Lustre: server umount lustre-MDT0000 complete [ 1085.283342] LustreError: 38056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790971793 with bad export cookie 2245389913144579641 [ 1085.286980] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1085.305931] LustreError: 38056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1085.631174] Lustre: server umount lustre-MDT0001 complete [ 1099.952461] Lustre: server umount lustre-OST0000 complete [ 1114.719538] Lustre: server umount lustre-OST0001 complete [ 1123.431581] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 16:10:30 (1790971830) [ 1141.203526] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1151.797416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1152.287981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1152.292468] Lustre: Skipped 7 previous similar messages [ 1155.504153] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1162.313560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1165.922298] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1168.165842] Lustre: 41284:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1174.219980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1181.001776] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1189.117666] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1195.559730] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1202.781389] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1207.365911] Lustre: 43160:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1208.441817] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1208.446547] Lustre: Skipped 1 previous similar message [ 1213.920903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1213.928265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1213.935300] Lustre: Skipped 3 previous similar messages [ 1217.974554] Lustre: server umount lustre-MDT0000 complete [ 1221.566415] LustreError: 40114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790971929 with bad export cookie 2245389913144580747 [ 1221.571900] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1221.578146] LustreError: 40114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1227.898822] Lustre: server umount lustre-MDT0001 complete [ 1236.659995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1247.651536] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1f293a3abb3a5ef2 [ 1247.666290] Lustre: MGC192.168.202.120@tcp: Connection restored to 0@lo (at 0@lo) [ 1252.697848] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1256.431632] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1260.796903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1265.788515] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1266.671858] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1266.676667] Lustre: Skipped 1 previous similar message [ 1274.594709] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1276.237931] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1282.609555] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1284.122553] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1292.271706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1294.631336] Lustre: server umount lustre-MDT0000 complete [ 1297.378900] LustreError: 44781:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1297.402525] LustreError: 44781:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [ 1298.443716] LustreError: 40116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790972006 with bad export cookie 2245389913144581874 [ 1298.462302] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1298.902980] Lustre: server umount lustre-MDT0001 complete [ 1313.656912] Lustre: server umount lustre-OST0000 complete [ 1328.167074] Lustre: server umount lustre-OST0001 complete [ 1337.485648] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 16:14:04 (1790972044) [ 1355.986842] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1368.563995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1374.019552] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1382.567883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1387.751957] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1391.243810] Lustre: 48924:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1397.898014] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1403.973650] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1413.182831] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1420.509263] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1427.971285] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1432.499165] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1445.357838] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.373915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1445.374253] Lustre: Skipped 11 previous similar messages [ 1445.406803] Lustre: Skipped 22 previous similar messages [ 1446.722271] Lustre: server umount lustre-MDT0000 complete [ 1450.931121] LustreError: 47756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790972158 with bad export cookie 2245389913144582672 [ 1450.949400] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1450.960271] LustreError: 47756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 1451.257874] Lustre: server umount lustre-MDT0001 complete [ 1464.877199] Lustre: server umount lustre-OST0000 complete [ 1479.627976] Lustre: server umount lustre-OST0001 complete [ 1490.158953] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 16:16:36 (1790972196) [ 1492.689825] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1494.558659] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 16:16:41 (1790972201) [ 1506.890082] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 1517.420269] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1539.854152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1540.129709] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1540.275521] Lustre: lustre-MDT0000: new disk, initializing [ 1540.388201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1544.847542] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1554.217684] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1561.591768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1561.937269] Lustre: lustre-OST0000: new disk, initializing [ 1561.952295] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1561.954608] Lustre: Skipped 1 previous similar message [ 1561.969139] Lustre: 55009:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1563.730798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1563.743665] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1563.782121] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1568.687122] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1577.338944] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1579.510918] Lustre: DEBUG MARKER: create test files [ 1614.821087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1618.450415] Lustre: server umount lustre-OST0000 complete [ 1622.774058] Lustre: server umount lustre-MDT0000 complete [ 1634.606736] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 1638.231592] Key type lgssc unregistered [ 1638.648399] LNet: 56638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1638.656161] LNetError: 56638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1638.675064] LNet: Removed LNI 192.168.202.120@tcp [ 1639.757173] Key type .llcrypt unregistered [ 1639.762195] Key type ._llcrypt unregistered [ 1671.964638] Key type ._llcrypt registered [ 1671.966802] Key type .llcrypt registered [ 1672.053134] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1673.388178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1673.443526] alg: No test for adler32 (adler32-zlib) [ 1674.500657] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 1674.850783] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 1676.495220] Key type lgssc registered [ 1677.853764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1685.428249] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1685.450834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1686.961808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1691.119187] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1698.460301] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1705.342093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1705.769615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1711.588391] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1711.921212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1711.995930] Lustre: 59395:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1712.020339] Lustre: 59395:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1712.030346] Lustre: 59395:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1712.042775] Lustre: 59395:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1712.055867] Lustre: 59395:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1712.072316] Lustre: 59395:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1718.908670] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1723.896897] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1728.954115] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1737.538864] Lustre: server umount lustre-OST0000 complete [ 1751.684970] Lustre: server umount lustre-MDT0000 complete [ 1761.500896] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 1764.131300] Key type lgssc unregistered [ 1764.531652] LNet: 61003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1764.549314] LNetError: 61003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.567469] LNet: Removed LNI 192.168.202.120@tcp [ 1765.380344] Key type .llcrypt unregistered [ 1765.382585] Key type ._llcrypt unregistered [ 1793.119746] Key type ._llcrypt registered [ 1793.122945] Key type .llcrypt registered [ 1793.210289] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1794.587383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1794.617297] alg: No test for adler32 (adler32-zlib) [ 1795.663631] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 1795.939682] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 1797.695946] Key type lgssc registered [ 1798.714152] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1805.883513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1805.933270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1807.491181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1812.009955] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1818.844969] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1826.974434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1827.569467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1832.693802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1832.888632] Lustre: 63829:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1832.921432] Lustre: 63829:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1832.944405] Lustre: 63829:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1832.969391] Lustre: 63829:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1832.988320] Lustre: 63829:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1833.006322] Lustre: 63829:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1837.259149] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1845.937311] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1851.439109] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1894.368364] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1894.379089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1894.409950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1896.329705] Lustre: server umount lustre-OST0000 complete [ 1900.864396] Lustre: server umount lustre-MDT0000 complete [ 1914.090147] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 1917.788902] Key type lgssc unregistered [ 1918.178341] LNet: 65410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1918.186043] LNetError: 65410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1919.206406] LNet: Removed LNI 192.168.202.120@tcp [ 1920.238131] Key type .llcrypt unregistered [ 1920.242882] Key type ._llcrypt unregistered [ 1941.939797] Key type ._llcrypt registered [ 1941.943505] Key type .llcrypt registered [ 1942.052138] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 1961.432430] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1962.657565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1962.772068] alg: No test for adler32 (adler32-zlib) [ 1963.863691] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 1964.190846] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 1965.911380] Key type lgssc registered [ 1967.253349] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2003.381690] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2003.428763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2004.661316] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2004.681984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2004.737319] Lustre: lustre-MDT0000: new disk, initializing [ 2004.862690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2004.886339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2009.478321] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2020.734131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2020.869196] Lustre: 68583:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2020.949656] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2020.954687] Lustre: Skipped 1 previous similar message [ 2021.042254] Lustre: lustre-MDT0001: new disk, initializing [ 2021.127928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2021.191877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2021.202934] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2025.488083] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2034.241344] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2041.199045] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2049.188254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2049.575958] Lustre: lustre-OST0000: new disk, initializing [ 2049.578583] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2049.585903] Lustre: 70551:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2049.731262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2055.212632] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2056.304713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2056.317614] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2056.412411] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2066.895888] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2072.282683] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2072.526634] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2077.595948] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2077.917393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2082.280735] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2082.291798] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2082.296221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2082.311623] Lustre: Skipped 1 previous similar message [ 2086.151312] Lustre: server umount lustre-OST0000 complete [ 2092.512679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2092.529863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2092.543799] Lustre: Skipped 2 previous similar messages [ 2097.632500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2097.640553] Lustre: Skipped 1 previous similar message [ 2102.768218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2102.783588] Lustre: Skipped 1 previous similar message [ 2104.288242] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2104.886757] Lustre: server umount lustre-MDT0000 complete [ 2109.746939] LustreError: 68575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790972817 with bad export cookie 14148183289320831 [ 2109.748368] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2109.776241] LustreError: 68575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2110.215329] Lustre: server umount lustre-MDT0001 complete [ 2122.205190] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 16:27:08 (1790972828) [ 2137.934509] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 2178.851609] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 2191.143133] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 2227.648743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2227.962376] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2227.991856] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2228.076497] Lustre: lustre-MDT0000: new disk, initializing [ 2228.145315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2228.153736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2233.406212] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2245.032629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2245.176396] Lustre: 76596:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2245.233806] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2245.237229] Lustre: Skipped 1 previous similar message [ 2245.340704] Lustre: lustre-MDT0001: new disk, initializing [ 2245.483621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2245.544043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2245.550677] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2252.161471] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2262.225428] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2269.880128] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2276.714040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2276.898227] Lustre: lustre-OST0000: new disk, initializing [ 2276.900800] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2276.906528] Lustre: 78563:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2276.972310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2282.158365] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2285.118643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2285.132212] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2285.233987] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2291.574350] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2296.248762] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2296.470934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2301.293568] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2301.528357] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2306.025033] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2306.025815] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2306.035769] Lustre: Skipped 1 previous similar message [ 2306.044778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2306.048373] LustreError: Skipped 2 previous similar messages [ 2306.061064] Lustre: Skipped 1 previous similar message [ 2310.027715] Lustre: server umount lustre-OST0000 complete [ 2316.257378] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2316.288636] Lustre: Skipped 1 previous similar message [ 2316.314485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2316.338587] Lustre: Skipped 1 previous similar message [ 2328.031163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2328.314929] Lustre: server umount lustre-MDT0000 complete [ 2331.625340] LustreError: 76606:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2331.652108] LustreError: 76606:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2332.322687] LustreError: 76586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790973040 with bad export cookie 14148183289321713 [ 2332.336580] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2332.648388] Lustre: server umount lustre-MDT0001 complete [ 2341.934962] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 16:30:48 (1790973048) [ 2349.392605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2350.080392] LustreError: 80977:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2350.175852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2355.143235] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2355.168498] LustreError: 80978:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2360.294254] LustreError: 80977:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2363.480472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2368.589891] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2375.889881] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2381.905223] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2388.997036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2389.296230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2389.301455] Lustre: Skipped 1 previous similar message [ 2394.731328] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2402.934176] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2407.177101] Lustre: 83648:0:(genops.c:1773:obd_export_evict_by_uuid()) MGS: evicting 8ffce570-3405-4a65-ae9e-873e29949384 at adminstrative request [ 2408.805867] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2408.829947] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2418.179564] Lustre: MGS: Client 8ffce570-3405-4a65-ae9e-873e29949384 (at 0@lo) reconnecting [ 2418.208184] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x3243b29f2d5c63 to 0x3243b29f2d5f03 [ 2418.248271] Lustre: MGC192.168.202.120@tcp: Connection restored to 0@lo (at 0@lo) [ 2445.024654] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2445.251207] Lustre: lustre-OST0001: new disk, initializing [ 2445.262393] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2445.271136] Lustre: 84042:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2453.032133] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2454.136470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2454.158576] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2454.281985] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2462.057612] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2469.858208] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2469.871275] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2469.883817] Lustre: Skipped 1 previous similar message [ 2469.889572] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2469.893611] Lustre: Skipped 5 previous similar messages [ 2479.073049] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2479.332200] Lustre: server umount lustre-OST0001 complete [ 2480.102792] LustreError: 82799:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2480.142399] LustreError: 82799:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2485.224333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2485.245376] Lustre: Skipped 2 previous similar messages [ 2490.361993] LustreError: 82798:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2490.375219] LustreError: 82798:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2490.475526] Lustre: server umount lustre-OST0000 complete [ 2500.743699] Lustre: server umount lustre-MDT0000 complete [ 2504.260404] LustreError: 80957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790973212 with bad export cookie 14148183289323267 [ 2504.267801] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2504.272708] LustreError: 80957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2504.756920] Lustre: server umount lustre-MDT0001 complete [ 2513.466609] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 16:33:40 (1790973220) [ 2514.850611] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2516.777590] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 16:33:43 (1790973223) [ 2518.307917] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2520.048336] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 16:33:47 (1790973227) [ 2521.778987] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2523.827883] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 16:33:50 (1790973230) [ 2525.582390] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2527.334624] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 16:33:54 (1790973234) [ 2529.036563] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2535.625824] loop: module loaded [ 2535.641865] loop0: detected capacity change from 0 to 4294967296 [ 2535.682576] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2537.534712] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2541.563991] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2546.106241] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2575.188532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2575.240579] LustreError: 88167:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2575.250456] LustreError: 88167:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2575.257514] LustreError: 88167:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2575.277428] LustreError: 88167:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2598.295160] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 2610.171197] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 2647.209273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2647.492404] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2647.522120] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2647.593206] Lustre: lustre-MDT0000: new disk, initializing [ 2647.688651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2647.691965] Lustre: Skipped 1 previous similar message [ 2647.727240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2652.373629] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2662.764494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2662.887484] Lustre: 91038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2663.120227] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2668.030823] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2677.442121] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2682.699911] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2689.300418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2689.597934] Lustre: 93005:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2694.774516] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2696.247038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2696.376735] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2703.231233] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2708.007251] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2708.169559] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2712.378727] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2712.536972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2716.641784] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2716.646573] LustreError: Skipped 1 previous similar message [ 2716.649499] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2716.657370] Lustre: Skipped 2 previous similar messages [ 2716.662392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2716.664860] Lustre: Skipped 8 previous similar messages [ 2720.351882] Lustre: server umount lustre-OST0000 complete [ 2724.832609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2724.838623] LustreError: Skipped 1 previous similar message [ 2729.909734] Lustre: server umount lustre-MDT0000 complete [ 2732.522903] LustreError: 91048:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2732.550119] LustreError: 91048:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2733.539493] LustreError: 91029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790973441 with bad export cookie 14148183289323883 [ 2733.569094] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2733.572075] LustreError: 91029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2733.988732] Lustre: server umount lustre-MDT0001 complete [ 2742.228747] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 16:37:29 (1790973449) [ 2747.926363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2752.835526] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2760.949404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2765.804762] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2773.426981] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2779.131653] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2785.838081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2786.225342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2786.234719] Lustre: Skipped 4 previous similar messages [ 2792.366890] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2800.643434] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2805.967644] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2806.174893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2811.561324] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2811.794701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2818.428532] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2818.580969] Lustre: lustre-OST0001: new disk, initializing [ 2818.583377] Lustre: Skipped 2 previous similar messages [ 2818.586982] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2818.590574] Lustre: Skipped 3 previous similar messages [ 2818.594505] Lustre: 98787:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2824.797485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2824.801467] Lustre: Skipped 2 previous similar messages [ 2824.818802] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2824.932990] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2825.069090] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2834.626172] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2839.565641] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2839.772606] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2844.280580] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2844.493629] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2861.323467] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2882.249981] Lustre: Failing over lustre-MDT0000 [ 2882.486654] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.20@tcp (stopping) [ 2882.497186] Lustre: Skipped 6 previous similar messages [ 2882.763502] Lustre: server umount lustre-MDT0000 complete [ 2884.064785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2884.077436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2884.085609] Lustre: Skipped 3 previous similar messages [ 2884.093759] LustreError: 96147:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2884.107028] LustreError: 96147:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2901.983162] Lustre: 66795:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790973594/real 1790973594] req@00000000a36c5a12 x1877970960789760/t0(0) o400->MGC192.168.202.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790973610 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2902.001164] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2908.526732] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2908.531986] LDISKFS-fs (dm-0): recovery complete [ 2908.542556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2912.228818] LustreError: 66794:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000067bc328f x1877970960798080/t0(0) o250->MGC192.168.202.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2913.039966] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2914.186450] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2914.188813] Lustre: 95414:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2918.372928] Lustre: 95420:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2918.401897] Lustre: 95420:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 1 previous similar message [ 2918.431918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2918.533091] LustreError: 101633:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2918.839458] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2918.881389] LustreError: 101633:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2919.220968] LustreError: 101633:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2919.239845] LustreError: 101633:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 1 previous similar message [ 2919.605124] LustreError: 101633:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2919.607633] LustreError: 101633:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 1 previous similar message [ 2919.629617] Lustre: 101633:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2919.651112] Lustre: 101633:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 7 previous similar messages [ 2920.282130] LustreError: 101633:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2920.287804] LustreError: 101633:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 2 previous similar messages [ 2920.608501] LustreError: 101633:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2920.616218] LustreError: 101633:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 2 previous similar messages [ 2921.897871] Lustre: 101633:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2921.916055] Lustre: 101633:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 6 previous similar messages [ 2922.583559] LustreError: 101633:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2922.596705] LustreError: 101633:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 6 previous similar messages [ 2922.919123] LustreError: 101633:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2922.923261] LustreError: 101633:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 6 previous similar messages [ 2926.114731] Lustre: 101633:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2926.138582] Lustre: 101633:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 12 previous similar messages [ 2926.776195] LustreError: 101633:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2926.794049] LustreError: 101633:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 12 previous similar messages [ 2927.119784] LustreError: 101633:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2927.128885] LustreError: 101633:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 12 previous similar messages [ 2934.117467] Lustre: 101633:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2934.140845] Lustre: 101633:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 24 previous similar messages [ 2934.792255] LustreError: 101633:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2934.802116] LustreError: 101633:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 24 previous similar messages [ 2935.127130] LustreError: 101633:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2935.136519] LustreError: 101633:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 24 previous similar messages [ 2950.153447] Lustre: 101633:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2950.195615] Lustre: 101633:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 49 previous similar messages [ 2950.841705] LustreError: 101633:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2950.865951] LustreError: 101633:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 49 previous similar messages [ 2951.183334] LustreError: 101633:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2951.202727] LustreError: 101633:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 49 previous similar messages [ 2974.079183] Lustre: 101633:0:(ldlm_lib.c:2428:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2974.086775] Lustre: 101633:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2974.098600] Lustre: 101633:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 1d50c936-c8d4-413f-9c9e-16f17691779b@192.168.202.20@tcp [ 2974.130050] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2974.150937] Lustre: 101633:0:(ldlm_lib.c:1948:abort_req_replay_queue()) @@@ aborted: req@000000005f8dc115 x1877970940660224/t0(8589934770) o35->1d50c936-c8d4-413f-9c9e-16f17691779b@192.168.202.20@tcp:442/0 lens 392/0 e 0 to 0 dl 1790973692 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2974.216713] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2974.250115] Lustre: lustre-MDT0000: Denying connection for new client 1d50c936-c8d4-413f-9c9e-16f17691779b (at 192.168.202.20@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2974.257293] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2974.347427] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2974.355264] Lustre: Skipped 2 previous similar messages [ 2974.361879] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2974.436977] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:45 to 0x2c0000401:545) [ 2974.437505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:78 to 0x280000401:577) [ 2992.097609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2992.100570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2992.120915] Lustre: Skipped 3 previous similar messages [ 2996.984364] Lustre: server umount lustre-OST0000 complete [ 3000.300542] LustreError: 99128:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3000.326699] LustreError: 99128:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 41 previous similar messages [ 3006.835554] Lustre: server umount lustre-OST0001 complete [ 3015.654227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3015.665081] Lustre: Skipped 8 previous similar messages [ 3025.375115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3025.481784] Lustre: server umount lustre-MDT0000 complete [ 3029.709687] LustreError: 95399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790973737 with bad export cookie 14148183289375095 [ 3029.712620] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3029.718191] LustreError: 95399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3030.114442] Lustre: server umount lustre-MDT0001 complete [ 3040.889892] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 16:42:27 (1790973747) [ 3049.900054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3049.908281] Lustre: *** cfs_fail_loc=197, val=0*** [ 3065.503798] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3070.624977] LustreError: 103611:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.120@tcp: failed processing log, type 4: rc = -110 [ 3075.795189] Lustre: *** cfs_fail_loc=197, val=0*** [ 3075.804515] Lustre: Skipped 65 previous similar messages [ 3096.355230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3096.358892] Lustre: Skipped 2 previous similar messages [ 3102.853432] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3112.039886] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3114.406478] Lustre: server umount lustre-OST0000 complete [ 3123.528076] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 16:43:50 (1790973830) [ 3146.521257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3147.199735] LustreError: 105231:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3147.228697] LustreError: 105231:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3153.600468] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3167.199152] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3171.454201] LustreError: 105262:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3171.463367] LustreError: 105262:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3171.473731] LustreError: 105262:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 24, retries 0, failed: rc = -5 [ 3171.973715] Lustre: server umount lustre-MDT0000 complete [ 3211.255094] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 3223.032914] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 3258.272590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3258.488987] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3258.516129] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3258.587842] Lustre: lustre-MDT0000: new disk, initializing [ 3258.722659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3263.469636] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3275.771717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3276.032440] Lustre: 109437:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3276.504226] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3282.532785] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3292.033589] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3298.142070] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3306.628916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3306.777520] Lustre: 111408:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3313.155629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3314.841395] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3327.026805] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3332.938385] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3333.135332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3337.470820] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3337.596566] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3343.843808] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3343.856549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3343.867192] LustreError: Skipped 4 previous similar messages [ 3343.872776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3343.890030] Lustre: Skipped 6 previous similar messages [ 3343.913911] Lustre: Skipped 3 previous similar messages [ 3345.796420] Lustre: server umount lustre-OST0000 complete [ 3363.295894] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3363.628258] Lustre: server umount lustre-MDT0000 complete [ 3368.199574] LustreError: 109429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790974076 with bad export cookie 14148183289385693 [ 3368.209636] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3368.216840] LustreError: 109429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3368.926907] Lustre: server umount lustre-MDT0001 complete [ 3378.870507] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 16:48:05 (1790974085) [ 3389.969062] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 3392.999731] Key type lgssc unregistered [ 3393.268591] LNet: 113894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3393.271787] LNetError: 113894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3393.280350] LNet: Removed LNI 192.168.202.120@tcp [ 3394.214790] Key type .llcrypt unregistered [ 3394.216615] Key type ._llcrypt unregistered [ 3413.037335] Key type ._llcrypt registered [ 3413.041329] Key type .llcrypt registered [ 3413.168453] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 3428.619975] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 3429.349588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3429.372897] alg: No test for adler32 (adler32-zlib) [ 3430.524670] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 3430.901912] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 3432.521214] Key type lgssc registered [ 3433.766604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3484.142431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3484.160202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3485.339675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3485.366751] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3485.427336] Lustre: lustre-MDT0000: new disk, initializing [ 3485.475139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3485.493277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3490.560510] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3502.779221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3503.115778] Lustre: lustre-OST0000: new disk, initializing [ 3503.119501] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3503.123111] Lustre: Skipped 1 previous similar message [ 3503.129107] Lustre: 118086:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3503.201375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3510.217633] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3512.442862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3512.451780] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3512.486819] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3520.558247] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3520.909591] Lustre: lustre-OST0001: new disk, initializing [ 3520.917051] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3520.934245] Lustre: 118950:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3521.093252] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3521.560567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3521.574697] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3521.621738] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3528.027970] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3536.865609] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3536.883148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3536.886639] Lustre: Skipped 1 previous similar message [ 3536.900621] Lustre: Skipped 1 previous similar message [ 3541.986301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3541.992877] Lustre: Skipped 1 previous similar message [ 3543.244895] Lustre: server umount lustre-MDT0000 complete [ 3550.451516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3557.155964] LustreError: 118949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790974265 with bad export cookie 1490736763141987983 [ 3557.166544] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3557.273963] Lustre: server umount lustre-OST0000 complete [ 3561.918656] Lustre: server umount lustre-OST0001 complete [ 3576.054175] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 16:51:22 (1790974282) [ 3599.989696] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 16:51:47 (1790974307) [ 3610.570097] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 3641.060667] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 16:52:27 (1790974347) [ 3682.938123] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 3694.689887] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 3735.108427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3735.302223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3735.324279] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3735.372493] Lustre: lustre-MDT0000: new disk, initializing [ 3735.435982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3735.448746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3741.247086] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3754.802564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3755.125744] Lustre: 126705:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3755.278635] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3755.289972] Lustre: Skipped 1 previous similar message [ 3755.468569] Lustre: lustre-MDT0001: new disk, initializing [ 3755.737213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3755.899160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3755.923788] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3762.929313] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3773.596075] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3780.485668] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3788.275338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3788.602440] Lustre: lustre-OST0000: new disk, initializing [ 3788.609396] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3788.619672] Lustre: 128676:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3788.810924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3796.485165] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3798.626099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3798.642397] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3798.802262] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3807.826645] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3813.718579] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3813.974537] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3819.895334] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3820.237931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3824.608355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3824.613142] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3824.614491] LustreError: Skipped 1 previous similar message [ 3824.624576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3824.637850] Lustre: Skipped 1 previous similar message [ 3824.664678] Lustre: Skipped 1 previous similar message [ 3828.811463] Lustre: server umount lustre-OST0000 complete [ 3834.852085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3834.868996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3834.874706] Lustre: Skipped 1 previous similar message [ 3838.941833] Lustre: server umount lustre-MDT0000 complete [ 3843.484630] LustreError: 126698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790974551 with bad export cookie 1490736763141989208 [ 3843.486457] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3843.509768] LustreError: 126698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3843.906531] Lustre: server umount lustre-MDT0001 complete [ 3853.660423] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 16:56:00 (1790974560) [ 3861.214275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3861.915193] LustreError: 131089:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3861.940276] LustreError: 131089:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3862.009502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3867.111067] LustreError: 131090:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3868.084657] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3871.204749] LustreError: 131089:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3876.326343] LustreError: 131090:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3878.579948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3884.989285] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3895.181613] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3901.580252] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3909.703454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3910.192214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3910.205398] Lustre: Skipped 1 previous similar message [ 3917.284949] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3925.573189] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3941.124711] Lustre: *** cfs_fail_loc=159, val=0*** [ 3956.730148] Lustre: lustre-MDT0000: Client ffd18a16-f8a5-4fd7-85ff-ac299f25cb7b (at 192.168.202.20@tcp) reconnecting [ 3958.748434] Lustre: *** cfs_fail_loc=159, val=0*** [ 3958.755212] Lustre: Skipped 5 previous similar messages [ 3975.228699] Lustre: lustre-MDT0000: Client ffd18a16-f8a5-4fd7-85ff-ac299f25cb7b (at 192.168.202.20@tcp) reconnecting [ 3993.569963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3993.580152] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3993.608679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3993.608688] Lustre: Skipped 1 previous similar message [ 3998.699957] LustreError: 132911:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3999.108870] Lustre: server umount lustre-OST0000 complete [ 4003.812164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4003.827389] Lustre: Skipped 1 previous similar message [ 4003.830389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4003.832668] Lustre: Skipped 1 previous similar message [ 4010.274242] Lustre: server umount lustre-MDT0000 complete [ 4014.053763] LustreError: 133581:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4014.086654] LustreError: 133581:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4015.888041] LustreError: 131070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790974723 with bad export cookie 1490736763141990090 [ 4015.892151] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4015.915572] LustreError: 131070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4016.495269] Lustre: server umount lustre-MDT0001 complete [ 4030.382968] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 4033.762690] Key type lgssc unregistered [ 4034.144550] LNet: 134888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4034.160385] LNetError: 134888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4034.187848] LNet: Removed LNI 192.168.202.120@tcp [ 4035.279622] Key type .llcrypt unregistered [ 4035.283560] Key type ._llcrypt unregistered [ 4065.347460] Key type ._llcrypt registered [ 4065.349020] Key type .llcrypt registered [ 4065.675731] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 4068.097465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4068.228657] alg: No test for adler32 (adler32-zlib) [ 4069.537187] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 4069.907488] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 4071.591286] Key type lgssc registered [ 4073.168384] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4081.557120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4081.585870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4083.222608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4089.505456] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4093.412963] LustreError: 136318:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4093.440770] LustreError: 136318:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4098.538087] LustreError: 136319:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4098.983417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4099.412625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4104.333410] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4112.690377] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4119.121916] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4126.549783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4126.895648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4128.955701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4134.536719] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4142.983069] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4153.569861] Lustre: *** cfs_fail_loc=159, val=0*** [ 4170.183210] Lustre: lustre-MDT0000: Client ed116701-3e7e-4f9b-8c91-139e068f625d (at 192.168.202.20@tcp) reconnecting [ 4172.751966] Lustre: *** cfs_fail_loc=159, val=0*** [ 4189.143848] Lustre: lustre-MDT0001: Client ed116701-3e7e-4f9b-8c91-139e068f625d (at 192.168.202.20@tcp) reconnecting [ 4191.051560] Lustre: *** cfs_fail_loc=159, val=0*** [ 4191.055881] Lustre: Skipped 3 previous similar messages [ 4207.560215] Lustre: lustre-MDT0001: Client ed116701-3e7e-4f9b-8c91-139e068f625d (at 192.168.202.20@tcp) reconnecting [ 4223.785498] Lustre: 139392:0:(mdt_handler.c:65:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4228.018292] Lustre: *** cfs_fail_loc=159, val=0*** [ 4228.032327] Lustre: Skipped 4 previous similar messages [ 4243.894525] Lustre: lustre-MDT0000: Client 0ed41ea2-9b96-4646-8687-2128366bba09 (at 192.168.202.20@tcp) reconnecting [ 4245.129640] Lustre: *** cfs_fail_loc=159, val=0*** [ 4245.134662] Lustre: Skipped 13 previous similar messages [ 4260.824824] Lustre: lustre-MDT0000: Client 0ed41ea2-9b96-4646-8687-2128366bba09 (at 192.168.202.20@tcp) reconnecting [ 4262.796630] Lustre: 139680:0:(mdt_handler.c:65:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4282.848570] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4282.858670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4282.874736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4282.880686] Lustre: Skipped 1 previous similar message [ 4282.896458] Lustre: Skipped 1 previous similar message [ 4287.968792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4287.972275] Lustre: Skipped 1 previous similar message [ 4293.089369] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4293.092270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4293.118012] Lustre: Skipped 1 previous similar message [ 4293.440315] Lustre: server umount lustre-OST0000 complete [ 4298.212523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4298.215065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4298.235046] Lustre: Skipped 1 previous similar message [ 4303.331538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4303.334540] Lustre: Skipped 1 previous similar message [ 4304.539167] Lustre: server umount lustre-MDT0000 complete [ 4308.450724] LustreError: 136318:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4308.483693] LustreError: 136318:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4308.985369] LustreError: 136300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790975017 with bad export cookie 9775965160699201060 [ 4308.995326] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4309.421366] Lustre: server umount lustre-MDT0001 complete [ 4323.190779] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 4327.130967] Key type lgssc unregistered [ 4327.525364] LNet: 140763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4327.535668] LNetError: 140763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4327.549980] LNet: Removed LNI 192.168.202.120@tcp [ 4328.634833] Key type .llcrypt unregistered [ 4328.638749] Key type ._llcrypt unregistered [ 4358.584509] Key type ._llcrypt registered [ 4358.587149] Key type .llcrypt registered [ 4358.717383] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 4360.826394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4360.847861] alg: No test for adler32 (adler32-zlib) [ 4362.080353] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 4362.515368] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 4364.311232] Key type lgssc registered [ 4366.094302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4375.007377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4375.031904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4376.716280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4381.546443] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4386.274984] LustreError: 142195:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4386.295935] LustreError: 142195:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4390.431802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4390.845729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4395.883516] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4404.290829] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4410.510191] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4418.079508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4418.466222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4421.559919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4426.086547] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4434.997114] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4435.898586] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4442.168406] Lustre: 144778:0:(mdt_handler.c:65:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4445.437436] Lustre: 144827:0:(mdt_handler.c:65:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4447.119256] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4448.268677] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4452.091611] Lustre: 144974:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4452.881245] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4453.882674] Lustre: 145082:0:(mdt_handler.c:65:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4461.541773] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4461.544668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4461.556406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4461.590610] Lustre: Skipped 1 previous similar message [ 4464.045760] Lustre: server umount lustre-OST0000 complete [ 4471.780638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4471.785971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4471.799180] Lustre: Skipped 1 previous similar message [ 4471.808837] Lustre: Skipped 2 previous similar messages [ 4474.858276] Lustre: server umount lustre-MDT0000 complete [ 4476.918426] LustreError: 144937:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4476.946816] LustreError: 144937:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4478.624658] LustreError: 142174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790975186 with bad export cookie 17391417861333214209 [ 4478.635385] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4478.644912] LustreError: 142174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4479.054534] Lustre: server umount lustre-MDT0001 complete [ 4490.814276] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 4494.702710] Key type lgssc unregistered [ 4495.039482] LNet: 146138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4495.045561] LNetError: 146138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4496.109503] LNet: Removed LNI 192.168.202.120@tcp [ 4496.987343] Key type .llcrypt unregistered [ 4496.989928] Key type ._llcrypt unregistered [ 4526.917761] Key type ._llcrypt registered [ 4526.919581] Key type .llcrypt registered [ 4527.068951] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 4528.501209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4528.532182] alg: No test for adler32 (adler32-zlib) [ 4529.723539] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 4530.026512] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 4531.744732] Key type lgssc registered [ 4533.371891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4542.623610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4542.674947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4544.044131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4548.995798] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4554.229910] LustreError: 147570:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4554.258724] LustreError: 147570:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4559.330663] LustreError: 147571:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4559.508402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4560.132388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4565.033178] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4573.001336] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4579.460550] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4587.474829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4587.997334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4589.045716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4595.331817] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4603.162280] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4605.428990] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4612.750608] Lustre: *** cfs_fail_loc=159, val=0*** [ 4612.770864] Lustre: Skipped 1 previous similar message [ 4628.899599] Lustre: lustre-MDT0000: Client d0d8af2f-d050-4a22-8c7e-7e72d108602a (at 192.168.202.20@tcp) reconnecting [ 4634.275024] Lustre: server umount lustre-OST0000 complete [ 4641.249081] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4641.251462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4641.272722] Lustre: Skipped 1 previous similar message [ 4641.293624] Lustre: Skipped 1 previous similar message [ 4646.369241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4646.375545] Lustre: Skipped 1 previous similar message [ 4648.531540] Lustre: server umount lustre-MDT0000 complete [ 4651.493484] LustreError: 149932:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4651.538843] LustreError: 149932:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4652.511442] Lustre: 146870:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790975344/real 1790975344] req@000000002c7f5978 x1877973651197056/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790975360 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4652.589216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4654.059895] LustreError: 147549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790975362 with bad export cookie 10841822352604815262 [ 4654.063602] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4654.091321] LustreError: 147549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4655.018389] Lustre: server umount lustre-MDT0001 complete [ 4670.929689] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 4673.136797] Key type lgssc unregistered [ 4673.353096] LNet: 151236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4673.365136] LNetError: 151236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4673.395793] LNet: Removed LNI 192.168.202.120@tcp [ 4674.236475] Key type .llcrypt unregistered [ 4674.239282] Key type ._llcrypt unregistered [ 4701.122319] Key type ._llcrypt registered [ 4701.126490] Key type .llcrypt registered [ 4701.318900] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 4702.987514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4703.027691] alg: No test for adler32 (adler32-zlib) [ 4704.179910] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 4704.550217] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 4706.271470] Key type lgssc registered [ 4708.311932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4718.181859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4718.209915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4719.771655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4724.893504] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4729.329773] LustreError: 152668:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4729.342487] LustreError: 152668:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4734.181841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4734.514522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4739.496941] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4747.353955] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4753.313562] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4761.051296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4761.700932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4764.864085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4769.686092] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4778.416573] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4779.121636] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4797.856434] Lustre: lustre-OST0000: evicting a6d7b07a-12a4-40ff-b6b3-266ad1789231 (at 192.168.202.20@tcp) by administrative request [ 4797.861626] Lustre: Skipped 1 previous similar message [ 4830.696343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4830.701552] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4830.716689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4830.727659] Lustre: Skipped 1 previous similar message [ 4835.288028] Lustre: server umount lustre-OST0000 complete [ 4840.928232] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4840.969201] Lustre: Skipped 1 previous similar message [ 4840.998809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4846.049104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4846.056555] Lustre: Skipped 2 previous similar messages [ 4850.146021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4850.157479] Lustre: Skipped 1 previous similar message [ 4854.239348] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4854.429862] Lustre: server umount lustre-MDT0000 complete [ 4855.271838] LustreError: 152667:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4857.771431] LustreError: 152647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790975565 with bad export cookie 2825188587307352097 [ 4857.773391] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4857.798097] LustreError: 152647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4857.990652] Lustre: server umount lustre-MDT0001 complete [ 4868.124511] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 4871.383627] Key type lgssc unregistered [ 4871.755641] LNet: 156734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4871.779098] LNetError: 156734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4871.802156] LNet: Removed LNI 192.168.202.120@tcp [ 4873.754244] Key type .llcrypt unregistered [ 4873.756314] Key type ._llcrypt unregistered [ 4962.668509] Key type ._llcrypt registered [ 4962.673517] Key type .llcrypt registered [ 4962.804550] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 4964.124204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4964.228340] alg: No test for adler32 (adler32-zlib) [ 4965.389142] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 4965.679896] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 4967.375908] Key type lgssc registered [ 4968.786183] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4978.131877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4978.167403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4979.910877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4985.079726] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4989.414559] LustreError: 159243:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4989.472393] LustreError: 159243:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4994.532832] LustreError: 159243:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4995.453587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4996.091344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5001.740200] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5011.187178] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5017.568786] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5025.708317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5026.038059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5032.971134] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5040.375660] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 5040.965874] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5049.591111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 5065.184767] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5065.195742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5065.211703] Lustre: Skipped 1 previous similar message [ 5065.215206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5067.629541] Lustre: server umount lustre-OST0000 complete [ 5072.864624] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5072.871255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5072.902745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5072.904867] Lustre: Skipped 1 previous similar message [ 5075.425140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5075.426047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5078.801299] Lustre: server umount lustre-MDT0000 complete [ 5080.548669] LustreError: 160497:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5080.592554] LustreError: 160497:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5083.166615] LustreError: 159224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790975791 with bad export cookie 7453273241803555749 [ 5083.188098] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5083.886615] Lustre: server umount lustre-MDT0001 complete [ 5096.440604] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 5101.442807] Key type lgssc unregistered [ 5102.184899] LNet: 162810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5102.199698] LNetError: 162810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5102.247617] LNet: Removed LNI 192.168.202.120@tcp [ 5103.292436] Key type .llcrypt unregistered [ 5103.296176] Key type ._llcrypt unregistered [ 5131.341039] Key type ._llcrypt registered [ 5131.344785] Key type .llcrypt registered [ 5131.500378] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 5133.110760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5133.248676] alg: No test for adler32 (adler32-zlib) [ 5134.444179] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 5134.802295] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 5136.561121] Key type lgssc registered [ 5137.997919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5154.578255] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 17:17:41 (1790975861) [ 5172.236118] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 5182.236526] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 5219.569966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5219.625660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5220.860386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5220.898419] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5220.972395] Lustre: lustre-MDT0000: new disk, initializing [ 5221.066730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5221.079816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5224.867155] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5234.975380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5235.252264] Lustre: 167090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5235.343522] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5235.349463] Lustre: Skipped 1 previous similar message [ 5235.474268] Lustre: lustre-MDT0001: new disk, initializing [ 5235.647677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5235.685672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5235.710119] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5240.156182] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5251.011672] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5258.066253] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5265.236478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5265.563888] Lustre: lustre-OST0000: new disk, initializing [ 5265.567412] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5265.573536] Lustre: 169057:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5265.657626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5273.215323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5273.252524] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5273.327338] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5276.846707] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5289.351858] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5304.292891] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5304.300511] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5304.308824] LustreError: Skipped 1 previous similar message [ 5304.337043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5304.358662] Lustre: Skipped 1 previous similar message [ 5306.253859] Lustre: server umount lustre-OST0000 complete [ 5312.480297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5312.487282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5312.496216] Lustre: Skipped 1 previous similar message [ 5312.503073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5314.531817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5314.536677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5316.984503] Lustre: server umount lustre-MDT0000 complete [ 5321.254596] LustreError: 167084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790976029 with bad export cookie 16018369091672759510 [ 5321.277922] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5321.284901] LustreError: 167084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5322.117600] Lustre: server umount lustre-MDT0001 complete [ 5336.184468] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 5339.444927] Key type lgssc unregistered [ 5339.618918] LNet: 170966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5339.632888] LNetError: 170966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5339.648615] LNet: Removed LNI 192.168.202.120@tcp [ 5340.209948] Key type .llcrypt unregistered [ 5340.211968] Key type ._llcrypt unregistered [ 5368.407450] Key type ._llcrypt registered [ 5368.411039] Key type .llcrypt registered [ 5368.568557] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 5369.701468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5369.903289] alg: No test for adler32 (adler32-zlib) [ 5371.189599] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 5371.613332] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 5373.399237] Key type lgssc registered [ 5374.901451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5383.261916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5383.302927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5384.889340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5390.412155] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5394.407258] LustreError: 172399:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5394.442646] LustreError: 172399:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5399.143034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5399.645530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5404.306304] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5411.155829] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5417.323988] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5425.204551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5425.980425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5429.082633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5433.878401] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 5441.507079] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5450.721902] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5450.774421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5450.797672] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5450.819909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.841415] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5452.321605] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5452.345592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5452.368532] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5452.376565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.397761] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5453.720628] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5453.737679] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5453.756978] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5453.760401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.768985] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5454.618409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.733235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.716510] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5456.718530] Lustre: Skipped 2 previous similar messages [ 5456.734585] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5456.747231] Lustre: Skipped 2 previous similar messages [ 5456.753370] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5456.764492] Lustre: Skipped 2 previous similar messages [ 5456.770245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.784383] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5456.804245] Lustre: Skipped 2 previous similar messages [ 5457.774173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.789590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.899503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.807774] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5460.811676] Lustre: Skipped 3 previous similar messages [ 5460.828215] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5460.836650] Lustre: Skipped 3 previous similar messages [ 5460.841087] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5460.849366] Lustre: Skipped 3 previous similar messages [ 5460.871122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.902986] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5460.913363] Lustre: Skipped 3 previous similar messages [ 5461.843102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.835712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.696257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.606078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.537843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.659355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.882641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.811849] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5468.818504] Lustre: Skipped 7 previous similar messages [ 5468.833950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5468.852061] Lustre: Skipped 7 previous similar messages [ 5468.856730] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5468.861884] Lustre: Skipped 7 previous similar messages [ 5468.866534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.958191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.975252] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5469.982918] Lustre: Skipped 8 previous similar messages [ 5470.935531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.812990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.734695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.701358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.584630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.538562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.466268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.846559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.736947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.836531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.991128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.146505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.499451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.840720] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5484.849959] Lustre: Skipped 14 previous similar messages [ 5484.901880] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5484.922136] Lustre: Skipped 14 previous similar messages [ 5484.938926] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5484.945199] Lustre: Skipped 14 previous similar messages [ 5484.948299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.109696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.131392] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5486.136563] Lustre: Skipped 14 previous similar messages [ 5487.305264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.597191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.571081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.704586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.918445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.298915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.554617] LustreError: 172419:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5494.574830] LustreError: 172419:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5494.578651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.976339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.120156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.414849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.748626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.958928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.044837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.447738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.561568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.660355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.810499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.977596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.912891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.319148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.506560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.568117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.660209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.526990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.631598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.394284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.347403] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5517.360068] Lustre: Skipped 27 previous similar messages [ 5517.402776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5517.413074] Lustre: Skipped 27 previous similar messages [ 5517.428611] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5517.434243] Lustre: Skipped 27 previous similar messages [ 5517.436400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.513639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.525122] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5518.532711] Lustre: Skipped 27 previous similar messages [ 5519.514436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.038835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.064771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.197483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.355676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.354245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.547562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.545076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.615713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.711818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.713987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.865327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.908300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.734266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.800826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.690900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.757340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.893371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.935357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.942327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.120930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.152965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.188649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.158897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.003984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.900741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.778462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.581304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.260929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.359771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.109189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.038894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.846936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.697896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.600972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.564945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.321479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.314647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.431129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.357740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.370629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.527473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.816032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.106413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.697124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.200202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.959229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.861617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.762951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.814287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.495806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.182023] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.948932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.658716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.013429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.157868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.339292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.343950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.142837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.953345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.969583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.066594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.060533] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5582.067716] Lustre: Skipped 63 previous similar messages [ 5582.094433] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5582.112587] Lustre: Skipped 63 previous similar messages [ 5582.117749] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5582.124330] Lustre: Skipped 63 previous similar messages [ 5582.135178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.165134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.175940] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5583.185111] Lustre: Skipped 63 previous similar messages [ 5583.831250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.622626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.632511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.389855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.351216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.412704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.275296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.251422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.198992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.170738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.958720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.055974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.865937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.994865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.932272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.001372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.924493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.859580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.773671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.743837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.747713] LustreError: 172419:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5602.765440] LustreError: 172419:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5602.770417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.746846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.667669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.723899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.765903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.718369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.694190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.544057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.561165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.645962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.672458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.626867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.466666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.444015] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.328219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.196315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.949675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.711695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.590149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.730684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.061191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.574739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.536218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.500078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.179528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.529569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.175587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.358603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.354242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.233363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.113365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.210324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.499204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.733877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.040680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.033652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.107878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.230729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.277542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.334176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.216844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.367926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.622080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.753638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.875610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.836857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.790219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.752561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.876959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.660278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.523743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.517637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.421666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.145269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.607442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.834753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.031214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.790428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.703547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.828603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.071243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.052334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.952591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.060858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.228610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.162159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.897692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.626547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.472806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.200911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.203716] LustreError: 172419:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5674.222739] LustreError: 172419:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5674.237501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.349732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.646173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.580292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.796117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.566704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.662160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.478527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.862565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.248592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.757135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.925104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.190690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.911718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.153049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.437865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.746721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.058833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.057842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.927910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.964472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.944930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.904898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.985486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.132869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.601098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.056854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.178942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.934910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.811635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.511708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.652391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.751430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.651193] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5710.660571] Lustre: Skipped 124 previous similar messages [ 5710.681074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5710.690247] Lustre: Skipped 124 previous similar messages [ 5710.692644] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5710.694974] Lustre: Skipped 124 previous similar messages [ 5710.697688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.762482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.779740] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5711.785840] Lustre: Skipped 124 previous similar messages [ 5712.703305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.712951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.729462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.772624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.933623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.813779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.680579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.539748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.619448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.563379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.748476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.505722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.651742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.493575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.535198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.488846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.411876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.293551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.302866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.172072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.963489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.821263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.800679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.638655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.384240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.278566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.253879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.120527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.013469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.232841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.496347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.931334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.047550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.246188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.476348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.359046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.046926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.484797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.710656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.259310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.427615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.263370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.129188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.774527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.972893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.995464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.159557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.140574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.879875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.849488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.745938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.946077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.750909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.702786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.786305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.787478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.824850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.872967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.644629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.593430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.538776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.708610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.762953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.760229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.704062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.593665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.549048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.637507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.399417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.619178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.621506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.621888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.410187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.310324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.312371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.186742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.023251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.854093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.835914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.742270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.377576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.480507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.453235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.182090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.013754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.311870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.628462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.600796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.587511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.565380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.329258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.044550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.873038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.898318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.030244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.265536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.146378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.191310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.900025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.664653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.342378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.136216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.930051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.802634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.628470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.393428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.260962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.250480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.006994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.889691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.661987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.449861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.369626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.264066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.957281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.837377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.670658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.722516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.541197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.580762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.452070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.445399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.320797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.159949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.094804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.158156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.063484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.129426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.856690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.092471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.089076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.271640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.124732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.950132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.923763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.894082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.737234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.585898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.487114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.393538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.302698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.175090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.206974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.088348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.092878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.079981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.031832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.000336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.876382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.797275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.896248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.700766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.620511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.549609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.318355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.268808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.384927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.223889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.280474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.251172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.204258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.208748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.277471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.326420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.390571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.388103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.194105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.062227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.934985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.941939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.879074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.816157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.943328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.137360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.350857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.232588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.472716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.549826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.810762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.693648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.473389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.469450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.282157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.285037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.185584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.977603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.764726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.609265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.659760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.680714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.497170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.438448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.381153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.520319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.442203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.442335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.712893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.718473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.835421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.771815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.562890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.341422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.381782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.238438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.128098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.065507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.176741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.112962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.213816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.203444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.259876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.221441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.129942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.921989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.936960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.896928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.042983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.990285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.092568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.095730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.081061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.003676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.877249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.891030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.698853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.751727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.840989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.029110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.128388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.246459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.260698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.508869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.616789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.577468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.696214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.818557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.945840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.782616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.948372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.824356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.976311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.246572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.416984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.548479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.854932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.857237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.741363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.692880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.666444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.629070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.544318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.454367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.482828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.525330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.647974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.896944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.614727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.587827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.293206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.253240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.391625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.275090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.358685] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5967.364365] Lustre: Skipped 263 previous similar messages [ 5967.382521] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5967.391257] Lustre: Skipped 263 previous similar messages [ 5967.395586] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5967.400348] Lustre: Skipped 263 previous similar messages [ 5967.404121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.437200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.447919] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5968.453187] Lustre: Skipped 263 previous similar messages [ 5969.826592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.895825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.942526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.215397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.157816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.087666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.908719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.886701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.145274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.186187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.236827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.101401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.145049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.152490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.207746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.276053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.095895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.985791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.183670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.287773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.183147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.361587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.373549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.183345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.158073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.217954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.274323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.506888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.681031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.919617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.058892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.154089] LustreError: 172419:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6002.163642] LustreError: 172419:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6002.169962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.738835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.961715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.124042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.361416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.491926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.395469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.868107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.991628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.977679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.921762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.690167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.538805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.823425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.816322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.928625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.220620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.234869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.546082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.463570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.592269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.847307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.976509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.999423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.016449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.011772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.185759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.522651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.810149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.506679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.567659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.293376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.205371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.893667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.145369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.252769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.919187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.894298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.879799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.945651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.639530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.632575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.654163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.151344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.700633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.854049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.464196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.788761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.425620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.597163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.177378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.651033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.643079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.892302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.461979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.907360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.460823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.445530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.313793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.421311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.329289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.414190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.376893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.418153] LustreError: 172419:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6074.429682] LustreError: 172419:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6074.443545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.251448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.377479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.298414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.573827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.533763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.706957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.771756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.678336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.619179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.693940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.461091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.329996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.172067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.225669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.124093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.085877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.071073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.990393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.016318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.983897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.065216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.064217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.178147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.377745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.549710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.569782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.810028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.770577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.888728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.849864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.160826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.081131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.087460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.046863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.121423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.096911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.947699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.876698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.850976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.923767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.966417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.928899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.825694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.708526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.621057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.456403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.372119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.162539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.177247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.149849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.120249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.050261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.110567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.057330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6129.211461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.178311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.190284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.040411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.100881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.250777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.196054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.232185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.217925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.441793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.341111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6140.484633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.394791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.453211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.377536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.501583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.623694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.598050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.874538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.732312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.773742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6150.630953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.700485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.527672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6153.496266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6154.266550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.426543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.485174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6157.422415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6158.704865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6160.096615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.394150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.857606] LustreError: 172419:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6162.868556] LustreError: 172419:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6162.891314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6164.433485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.074709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6167.764142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6169.261681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6170.681350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6171.707927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6172.513740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.130541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6175.786839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.020520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6178.166058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6179.093613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6180.062536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6181.119441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6182.152876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.294102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.328760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6185.596655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6186.779510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.800940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.971556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6189.921663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.871622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6192.061390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6193.369424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.430595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.412922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.702721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.630936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.665190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6199.806378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6200.925671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6202.138508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.010056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6204.090050] LustreError: 172419:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6204.102043] LustreError: 172419:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6204.117453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6204.939441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6205.846527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.631951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6207.577701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6208.532091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6209.498860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6210.519759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6211.570859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6212.712669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6213.718524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6214.947896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6215.852171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6217.074084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6218.303188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6219.432409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6220.450499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6221.483138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6222.539092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6223.563636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6224.548752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6225.618290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6226.743701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6228.007421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6228.924453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6230.027617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6231.050254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6232.085893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6233.125113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6234.232849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6235.169621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6236.211269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6237.076223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6238.190505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6239.383189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6240.322620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6241.564520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6242.492848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6243.511293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6244.237945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6245.024818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6245.761913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6246.694851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6247.359835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6248.411456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6249.225300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6250.175452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6251.171040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6252.616022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6254.180227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6255.700877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6257.112884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6257.909366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6258.925027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6259.577453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6260.496505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6261.482756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6262.575253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6263.608503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6265.015264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6266.420492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6267.345134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6268.303120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6269.896298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6271.354720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6272.423588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6273.656083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6275.071211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6276.614365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6277.865213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6279.067257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6280.106462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6281.253302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6282.463153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6283.369919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6284.411876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6285.524813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6286.780081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6287.894983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6289.026865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6290.150616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6291.093221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6291.711321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6292.475376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6293.548478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6294.661614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6295.668774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6296.580069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6297.610854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6298.639228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6299.799050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6300.587721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6301.737690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6303.083042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6304.403800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6305.746585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6306.476382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6307.450077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6308.150188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6309.742310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6310.796934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6311.544984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6312.301782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6313.396570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6314.458713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6315.374543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6316.418243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6317.510834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6318.895574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6320.065985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6321.099971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6321.894949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6322.925074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6324.060193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6325.044851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6326.385979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6327.885484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6329.157746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6330.232958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6331.289695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6332.293902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6333.402260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6334.304263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6335.458349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6336.651128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6337.709054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6339.105094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6340.355441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6341.360387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6342.477565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6343.723549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6344.801426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6345.841368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6347.081698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6348.063570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6349.264627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6350.298946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6351.512721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6352.597717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6353.751783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6354.663376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6355.561896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6356.476962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6357.532264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6358.345413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6359.237778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6360.176602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6361.239272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6362.033797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6363.073835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6364.052928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6365.013137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6365.928723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6366.926525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6367.817718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6368.899180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6370.123689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6371.037025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6372.035248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6373.134203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6374.152569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6375.216886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6376.110917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6377.280744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6378.182978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6379.106370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6379.967034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6381.006236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6381.898539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6382.804596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6383.845762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6384.771942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6385.970871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6387.157682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6388.066137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6389.157402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6390.238662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6391.449576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6392.404451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6393.317797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6394.320755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6395.373598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6396.299066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6397.379969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6398.505416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6399.453837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6400.423714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6401.317389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6402.546807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6403.477699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6404.825269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6406.142583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6406.956953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6408.168819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6409.008915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6410.058592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6411.168250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6412.317748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6413.304090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6414.372621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6415.356756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6416.503091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6417.484161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6418.572576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6419.300446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6420.358774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6421.222825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6421.999581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6422.715243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6423.650795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6424.310840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6425.098118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6425.907481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6426.997064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6427.789496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6429.193993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6430.583302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6432.215316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6433.503943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6434.685370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6435.802462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6437.034946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6437.999179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6439.231781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6440.234257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6441.508258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6442.699864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6443.595569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6444.776480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6445.867758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6447.057042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6447.945347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6448.894279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6449.905715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6450.969890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6451.983063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6453.151254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6454.142165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6455.262341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6456.413809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6457.451544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6458.767296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6459.876789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6460.761563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6462.030878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6462.983547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6464.051258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6465.173307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6466.288168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6467.522090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6468.537503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6469.735283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6470.761698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6471.691345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6472.631613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6473.734152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6474.592692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6475.575607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6476.601757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6477.742621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6478.876488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6479.824978] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6479.831957] Lustre: Skipped 478 previous similar messages [ 6479.848593] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6479.860666] Lustre: Skipped 478 previous similar messages [ 6479.871830] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6479.878746] Lustre: Skipped 478 previous similar messages [ 6479.881603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6480.945269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6480.956503] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6480.962704] Lustre: Skipped 478 previous similar messages [ 6482.001584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6483.252201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6484.332958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6485.299594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6486.291708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6487.393620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6862.366367] LustreError: 174222:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6862.372887] LustreError: 172419:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 7497.190696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7497.201456] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7497.217423] Lustre: Skipped 7 previous similar messages [ 7497.225609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7502.208339] Lustre: server umount lustre-OST0000 complete [ 7507.940387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7507.948283] Lustre: Skipped 2 previous similar messages [ 7513.062087] LustreError: 226495:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7513.084383] LustreError: 226495:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7513.335731] Lustre: server umount lustre-MDT0000 complete [ 7517.173939] LustreError: 172395:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7517.198246] LustreError: 172395:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7518.538281] LustreError: 172381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790978226 with bad export cookie 12036037754391681400 [ 7518.571817] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7519.499690] Lustre: server umount lustre-MDT0001 complete [ 7532.784368] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 7536.084713] Key type lgssc unregistered [ 7536.408953] LNet: 227658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7536.414206] LNetError: 227658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7536.432249] LNet: Removed LNI 192.168.202.120@tcp [ 7537.481685] Key type .llcrypt unregistered [ 7537.492606] Key type ._llcrypt unregistered [ 7565.884672] Key type ._llcrypt registered [ 7565.891947] Key type .llcrypt registered [ 7566.020931] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 7567.235832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7567.466767] alg: No test for adler32 (adler32-zlib) [ 7568.590490] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 7568.903800] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 7570.673843] Key type lgssc registered [ 7572.335852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7581.748764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7581.798590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7583.384548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7587.771179] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7593.959489] LustreError: 229090:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7593.990863] LustreError: 229090:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7596.889391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7597.327222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7603.135751] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7613.146244] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7619.432245] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7628.176658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7628.835875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7635.526459] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7643.551205] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7653.441386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7660.261240] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7661.854995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7661.866735] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7661.873013] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7661.886249] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7661.889492] LustreError: 229109:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7666.091253] Lustre: lustre-OST0000: Not available for connect from 192.168.202.20@tcp (stopping) [ 7667.681542] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7667.689240] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7667.709456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7668.706655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7668.710028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7668.724145] Lustre: Skipped 1 previous similar message [ 7670.262748] Lustre: server umount lustre-OST0000 complete [ 7681.936716] LustreError: 229109:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7706.749361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7707.089277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7708.166951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7708.200137] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7714.469427] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7718.440486] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7718.466617] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7722.903448] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7732.653417] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 7 sec [ 7739.342521] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff95c1075b2000.ost_server_uuid 50 [ 7740.642573] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff95c1075b2000.ost_server_uuid in FULL state after 0 sec [ 7748.718332] Lustre: server umount lustre-OST0000 complete [ 7756.262932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7756.291627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7756.321423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7764.448673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7764.451230] Lustre: Skipped 2 previous similar messages [ 7765.471169] Lustre: 228392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790978457/real 1790978457] req@000000005079fa52 x1877976836830720/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790978473 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7765.472431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7765.477806] LustreError: 229109:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000aaaab39c x1877976836834688/t0(0) o7->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 264/240 e 0 to 0 dl 0 ref 2 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7765.527711] Lustre: 228392:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7765.587987] Lustre: Skipped 1 previous similar message [ 7765.665212] Lustre: server umount lustre-MDT0000 complete [ 7770.284303] LustreError: 230933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790978478 with bad export cookie 17959197328424631564 [ 7770.299614] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7770.303398] LustreError: 230933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7770.846837] Lustre: server umount lustre-MDT0001 complete [ 7783.692386] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 7786.402585] Key type lgssc unregistered [ 7786.701733] LNet: 234337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7786.709154] LNetError: 234337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7786.737419] LNet: Removed LNI 192.168.202.120@tcp [ 7787.654356] Key type .llcrypt unregistered [ 7787.656637] Key type ._llcrypt unregistered [ 7825.741539] Key type ._llcrypt registered [ 7825.743518] Key type .llcrypt registered [ 7825.891369] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 7827.736035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7827.829793] alg: No test for adler32 (adler32-zlib) [ 7829.063714] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 7829.472025] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 7831.279257] Key type lgssc registered [ 7832.657357] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7842.706948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7842.725896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7844.257700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7848.262257] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7854.580931] LustreError: 236083:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7854.613606] LustreError: 236083:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7857.824677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7858.271918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7864.595415] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7868.262790] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7875.372402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7875.915521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7883.279805] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7889.310012] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7893.227199] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7893.404952] Lustre: lustre-OST0001: new disk, initializing [ 7893.409075] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7893.422579] Lustre: 238392:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7893.530572] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7900.446077] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7903.855361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7903.868336] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7904.015625] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7911.854519] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7923.322546] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7940.064841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7940.073549] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7940.087339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7940.103143] Lustre: Skipped 3 previous similar messages [ 7945.185674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7945.195877] Lustre: Skipped 5 previous similar messages [ 7945.544091] Lustre: server umount lustre-MDT0000 complete [ 7949.827391] LustreError: 236065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790978657 with bad export cookie 9466342325238552472 [ 7949.833390] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7949.848668] LustreError: 236065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7950.187698] Lustre: server umount lustre-MDT0001 complete [ 7964.605263] Lustre: server umount lustre-OST0000 complete [ 7966.691920] Lustre: 235338:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790978658/real 1790978658] req@ffff97ace3600e00 x1877977110511360/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790978674 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7966.733206] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7968.988330] Lustre: server umount lustre-OST0001 complete [ 7995.945700] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 8007.713523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8008.172942] LustreError: 241506:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8008.265884] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8013.050284] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8013.281781] LustreError: 241507:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8018.405368] LustreError: 241506:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8022.578839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8023.249350] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8028.820205] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8032.264537] Lustre: 242647:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8041.431746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8041.898054] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8048.046059] LustreError: 243000:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8048.086443] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:100 to 0x280000400:129) [ 8048.986058] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8053.197083] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49760) [ 8058.343616] LustreError: 243000:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8058.379442] LustreError: 243000:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8059.685460] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8068.402393] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8077.182205] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8081.462749] Lustre: 244518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8094.692407] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8094.719482] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8094.751500] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8100.275988] Lustre: server umount mylustre-MDT0000 complete [ 8101.350703] LustreError: 241506:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8101.380762] LustreError: 241506:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8104.961952] LustreError: 241483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790978812 with bad export cookie 9466342325238555335 [ 8104.969946] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8104.980442] LustreError: 241483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8105.313324] Lustre: server umount mylustre-MDT0001 complete [ 8120.889393] Lustre: server umount mylustre-OST0000 complete [ 8122.847129] Lustre: 235336:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790978814/real 1790978814] req@ffff97accfd18e00 x1877977110577792/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790978830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8122.920151] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8122.938450] Lustre: Skipped 3 previous similar messages [ 8125.725523] Lustre: server umount mylustre-OST0001 complete [ 8151.875854] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 8163.689898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8164.230980] LustreError: 246828:0:(ldlm_lib.c:1199:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8164.317163] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8164.321914] Lustre: Skipped 1 previous similar message [ 8169.178702] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8178.226627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8183.653189] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8187.358356] Lustre: 247971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8195.456563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8203.070597] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:100 to 0x280000400:161) [ 8203.738338] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8210.427815] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49792) [ 8214.802690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8220.162103] Lustre: tfs-OST0001: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:65) [ 8222.302799] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8230.420646] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8233.967976] Lustre: 249839:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8240.614596] Lustre: tfs-MDT0000-lwp-OST0000: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8240.619560] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 8240.634452] Lustre: Skipped 2 previous similar messages [ 8240.650029] Lustre: Skipped 6 previous similar messages [ 8245.217308] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8245.233930] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 8246.801798] Lustre: server umount tfs-MDT0000 complete [ 8250.857632] LustreError: 247827:0:(ldlm_lib.c:1199:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8250.882135] LustreError: 247827:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 8251.231242] LustreError: 246804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790978959 with bad export cookie 9466342325238557288 [ 8251.242092] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8251.248479] LustreError: 246804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8251.905754] Lustre: server umount tfs-MDT0001 complete [ 8266.447766] Lustre: server umount tfs-OST0000 complete [ 8280.266819] Lustre: server umount tfs-OST0001 complete [ 8304.567848] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 8315.612634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8316.090619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8316.094784] Lustre: Skipped 3 previous similar messages [ 8320.650055] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8329.519918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8335.429667] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8338.422147] Lustre: 253234:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8346.337032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8354.407896] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8355.069908] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:100 to 0x280000400:193) [ 8361.482500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49824) [ 8364.389085] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8370.206535] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:97) [ 8372.448660] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8381.265779] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8385.279845] Lustre: 255105:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8393.153935] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 18:11:39 (1790979099) [ 8412.135640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8412.154314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8412.172740] Lustre: Skipped 1 previous similar message [ 8412.179916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8412.188531] Lustre: Skipped 3 previous similar messages [ 8416.597945] Lustre: server umount lustre-MDT0000 complete [ 8419.306570] LustreError: 252087:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8419.334971] LustreError: 252087:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 8420.715682] LustreError: 252069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790979128 with bad export cookie 9466342325238559206 [ 8420.717660] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8420.725644] LustreError: 252069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8430.580232] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8430.586936] Lustre: Skipped 9 previous similar messages [ 8435.167943] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8435.609864] Lustre: server umount lustre-MDT0001 complete [ 8454.113553] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8454.207180] Lustre: server umount lustre-OST0000 complete [ 8464.881493] Lustre: server umount lustre-OST0001 complete [ 8473.437924] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 8486.439612] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 8532.134761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8532.614153] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8532.640749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8532.744966] Lustre: lustre-MDT0000: new disk, initializing [ 8532.947916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 8532.951032] Lustre: Skipped 3 previous similar messages [ 8532.988388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8538.028898] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8549.693705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8549.829192] Lustre: 258295:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8549.886751] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8549.893060] Lustre: Skipped 1 previous similar message [ 8549.976873] Lustre: lustre-MDT0001: new disk, initializing [ 8550.133790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8550.155866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8555.265654] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8567.796507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8568.008754] Lustre: lustre-OST0000: new disk, initializing [ 8568.012628] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8568.018889] Lustre: 259879:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8569.238052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8569.250791] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8569.340802] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8576.211391] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8589.073260] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8589.196844] Lustre: lustre-OST0001: new disk, initializing [ 8589.201373] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8589.208828] Lustre: 260749:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8591.286550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8591.301668] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8591.481313] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8599.719488] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8616.013490] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 18:15:22 (1790979322) [ 8634.965906] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 18:15:41 (1790979341) [ 8651.233600] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8651.238697] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8651.250060] LustreError: Skipped 1 previous similar message [ 8651.251284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8651.292415] Lustre: Skipped 6 previous similar messages [ 8656.719866] Lustre: server umount lustre-OST0000 complete [ 8663.014329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8666.838767] Lustre: server umount lustre-MDT0000 complete [ 8670.673462] LustreError: 259089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790979378 with bad export cookie 9466342325238560984 [ 8670.675679] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8670.684028] LustreError: 259089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8677.067863] Lustre: server umount lustre-MDT0001 complete [ 8691.878868] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 8768.168479] Lustre: server umount lustre-OST0001 complete [ 8775.394335] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 8794.258537] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 8837.826344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8838.081761] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8838.111432] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8838.197736] Lustre: lustre-MDT0000: new disk, initializing [ 8838.281539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8838.285657] Lustre: Skipped 3 previous similar messages [ 8838.302700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8843.476230] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8854.122033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8854.283367] Lustre: 265609:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8854.394573] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8854.397359] Lustre: Skipped 1 previous similar message [ 8854.500263] Lustre: lustre-MDT0001: new disk, initializing [ 8854.750252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8854.761097] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8860.405640] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8870.831648] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8877.001523] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8884.413107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8884.649353] Lustre: lustre-OST0000: new disk, initializing [ 8884.653992] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8884.663835] Lustre: 267575:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8886.366209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8886.388639] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8886.534404] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8891.809591] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8902.475393] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8912.353179] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8912.359047] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8912.360314] LustreError: Skipped 1 previous similar message [ 8912.361269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8912.361274] Lustre: Skipped 8 previous similar messages [ 8912.391988] Lustre: Skipped 5 previous similar messages [ 8916.434921] Lustre: server umount lustre-OST0000 complete [ 8926.940857] Lustre: server umount lustre-MDT0000 complete [ 8927.714436] LustreError: 265621:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8927.758179] LustreError: 265621:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 16 previous similar messages [ 8930.638083] LustreError: 265602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790979638 with bad export cookie 9466342325238562454 [ 8930.640217] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8930.651632] LustreError: 265602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8930.991908] Lustre: server umount lustre-MDT0001 complete [ 8943.946912] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 8946.830239] Key type lgssc unregistered [ 8947.167665] LNet: 269337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8947.177169] LNetError: 269337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8947.194831] LNet: Removed LNI 192.168.202.120@tcp [ 8948.085432] Key type .llcrypt unregistered [ 8948.091232] Key type ._llcrypt unregistered [ 8975.355461] Key type ._llcrypt registered [ 8975.358429] Key type .llcrypt registered [ 8975.489693] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 8976.629602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8976.781569] alg: No test for adler32 (adler32-zlib) [ 8977.873446] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 8978.082776] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 8979.743210] Key type lgssc registered [ 8980.877539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8989.405197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8989.420244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8990.776666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8995.736893] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8999.974281] LustreError: 270819:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8999.986730] LustreError: 270819:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9005.028326] LustreError: 270820:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9005.357269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9005.791051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9010.232247] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9019.156068] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9027.172613] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9034.457176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9034.953594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9036.044474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 9042.578873] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9050.282340] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9051.271586] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 9055.727991] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9055.732236] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9055.759543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9055.772052] Lustre: Skipped 1 previous similar message [ 9055.785099] Lustre: Skipped 1 previous similar message [ 9059.228673] Lustre: server umount lustre-OST0000 complete [ 9065.955397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9065.961116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9065.995447] Lustre: Skipped 1 previous similar message [ 9066.018036] Lustre: Skipped 1 previous similar message [ 9071.084210] LustreError: 272108:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9071.122664] LustreError: 272108:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9071.833208] Lustre: server umount lustre-MDT0000 complete [ 9075.172430] LustreError: 270814:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9075.193262] LustreError: 270814:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9075.598812] LustreError: 271553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790979783 with bad export cookie 13862482773792489749 [ 9075.612980] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9075.613703] LustreError: 271553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9075.967831] Lustre: server umount lustre-MDT0001 complete [ 9086.897063] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 9089.835857] Key type lgssc unregistered [ 9090.211661] LNet: 274402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9090.220039] LNetError: 274402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9090.228548] LNet: Removed LNI 192.168.202.120@tcp [ 9091.401160] Key type .llcrypt unregistered [ 9091.405733] Key type ._llcrypt unregistered [ 9110.668950] Key type ._llcrypt registered [ 9110.670819] Key type .llcrypt registered [ 9110.752821] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 9112.014397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9112.037593] alg: No test for adler32 (adler32-zlib) [ 9113.106790] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 9113.328456] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 9115.007229] Key type lgssc registered [ 9115.965735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9124.685237] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9124.707917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9140.255525] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9145.375318] LustreError: 275610:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.120@tcp: failed processing log, type 4: rc = -110 [ 9172.063269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9179.202424] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9185.927195] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9188.244077] Lustre: server umount lustre-OST0000 complete [ 9206.976980] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 9210.008365] Key type lgssc unregistered [ 9210.255723] LNet: 277241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9210.263207] LNetError: 277241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9211.307496] LNet: Removed LNI 192.168.202.120@tcp [ 9212.240185] Key type .llcrypt unregistered [ 9212.248338] Key type ._llcrypt unregistered [ 9256.655546] Key type ._llcrypt registered [ 9256.661733] Key type .llcrypt registered [ 9256.779817] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 9258.327687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9258.423076] alg: No test for adler32 (adler32-zlib) [ 9259.352218] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 9259.509532] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 9261.191748] Key type lgssc registered [ 9262.564399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9277.131957] loop0: detected capacity change from 0 to 204800000 [ 9277.420595] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9277.429879] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9277.657481] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9278.662203] loop0: detected capacity change from 0 to 204800000 [ 9278.882613] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9278.922421] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9279.368715] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9280.403927] loop0: detected capacity change from 0 to 204800000 [ 9280.561565] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9280.578296] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9280.666435] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9281.783397] loop0: detected capacity change from 0 to 204800000 [ 9282.020937] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9283.366313] loop0: detected capacity change from 0 to 204800000 [ 9283.404275] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9287.862919] loop0: detected capacity change from 0 to 204800000 [ 9287.911367] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9292.086207] loop0: detected capacity change from 0 to 204800000 [ 9292.139830] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9296.592667] loop0: detected capacity change from 0 to 204800000 [ 9296.649707] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9301.170672] loop0: detected capacity change from 0 to 204800000 [ 9301.238409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9301.255030] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9301.334985] Lustre: 280676:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9301.347305] Lustre: 280676:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 9301.352918] Lustre: 280676:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 9301.359445] Lustre: 280676:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9301.366341] Lustre: 280676:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 9301.372482] Lustre: 280676:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9301.379678] Lustre: 280676:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9301.387719] Lustre: 280676:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 9301.394342] Lustre: 280676:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 9301.406708] Lustre: 280676:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 9301.429107] Lustre: 280676:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 9301.450485] Lustre: 280676:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 9301.464750] Lustre: 280676:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 9301.483830] Lustre: 280676:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 9306.969924] Lustre: server umount MGS complete [ 9307.858879] loop0: detected capacity change from 0 to 204800000 [ 9307.961566] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9308.566419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9308.623286] Lustre: lustre-MDT0000: Aborting client recovery [ 9309.407510] loop1: detected capacity change from 0 to 204800000 [ 9309.501599] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9309.608342] Lustre: 281108:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9309.629626] Lustre: 281108:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 9309.646036] Lustre: 281108:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9309.655894] Lustre: 281108:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 9309.672612] Lustre: 281108:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 9309.683530] Lustre: 281108:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9309.694410] Lustre: 281108:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9309.730806] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9309.918276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9309.994486] Lustre: lustre-MDT0001: Aborting client recovery [ 9310.922114] loop2: detected capacity change from 0 to 204800000 [ 9311.165446] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9311.273981] Lustre: 281190:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9311.290763] Lustre: 281190:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9311.301186] Lustre: 281190:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9311.310641] Lustre: 281190:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9311.320760] Lustre: 281190:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9311.334622] Lustre: 281190:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9311.344792] Lustre: 281190:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9311.499531] Lustre: Found index 0 for lustre-OST0000, updating log [ 9311.606468] Lustre: 281208:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9311.681321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9311.701392] Lustre: lustre-OST0000: Aborting recovery [ 9312.588685] loop3: detected capacity change from 0 to 204800000 [ 9312.766750] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9312.827193] Lustre: 281273:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9312.839139] Lustre: 281273:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9312.853278] Lustre: 281273:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9312.861758] Lustre: 281273:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9312.869933] Lustre: 281273:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9312.880700] Lustre: 281273:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9312.887826] Lustre: 281273:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9312.908411] Lustre: Found index 1 for lustre-OST0001, updating log [ 9312.949121] Lustre: 281279:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9313.011857] Lustre: lustre-OST0001: Aborting recovery [ 9318.129942] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9318.131031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9318.155874] LustreError: 281052:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 9318.191885] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 9318.205736] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 9319.166177] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 9319.169931] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 9322.190618] LustreError: 281019:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 7256us: rc = -2 [ 9322.715554] LustreError: 281017:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 1330us: rc = -2 [ 9322.743286] LustreError: 281017:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 21 previous similar messages [ 9335.776930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9335.785982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9335.803483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9339.876837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9339.877497] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9339.890572] Lustre: Skipped 1 previous similar message [ 9342.036802] Lustre: server umount lustre-MDT0000 complete [ 9342.885684] LustreError: 281003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790980050 with bad export cookie 1406082792203793714 [ 9342.903772] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9344.993210] LustreError: 281559:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9344.996227] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9345.001275] LustreError: 281559:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9345.009278] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9345.014058] Lustre: Skipped 2 previous similar messages [ 9345.016817] Lustre: Skipped 3 previous similar messages [ 9348.000317] LustreError: 281559:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9348.011695] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9348.023948] LustreError: 281559:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9348.039894] Lustre: Skipped 1 previous similar message [ 9351.071960] LustreError: 281559:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9351.088569] LustreError: 281559:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9352.101252] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9352.114367] Lustre: Skipped 3 previous similar messages [ 9357.279165] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9357.282760] LustreError: 281017:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9357.320604] LustreError: 281017:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9357.678121] Lustre: server umount lustre-MDT0001 complete [ 9364.808672] Lustre: server umount lustre-OST0000 complete [ 9371.966684] Lustre: server umount lustre-OST0001 complete [ 9380.052275] Lustre: DEBUG MARKER: == conf-sanity test 108c: mkfs.lustre --mmp option enables MMP feature ========================================================== 18:28:06 (1790980086) [ 9404.911954] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 9447.341120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9447.514638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9447.550252] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9447.602150] Lustre: lustre-MDT0000: new disk, initializing [ 9447.730605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9447.734521] Lustre: Skipped 1 previous similar message [ 9447.744385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9453.416551] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9461.096381] Lustre: server umount lustre-MDT0000 complete [ 9479.575728] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 18:29:46 (1790980186) [ 9504.710054] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 9514.893704] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 9554.218720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9554.550472] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9554.573384] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9554.576731] Lustre: Skipped 1 previous similar message [ 9554.628089] Lustre: lustre-MDT0000: new disk, initializing [ 9554.679935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9554.698590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9560.079085] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9571.931335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9572.120503] Lustre: 288386:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9572.171519] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9572.174613] Lustre: Skipped 1 previous similar message [ 9572.269789] Lustre: lustre-MDT0001: new disk, initializing [ 9572.446539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9572.487820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9572.507984] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9577.880593] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9588.025110] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9594.612261] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9602.078666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9602.398963] Lustre: lustre-OST0000: new disk, initializing [ 9602.404984] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9602.411220] Lustre: 290356:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9602.545549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9608.942499] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9611.345308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9611.351983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9611.471458] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9617.899486] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9622.708612] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9635.882871] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9643.449604] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9655.101650] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9698.804216] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9698.806133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9698.821839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9698.833503] Lustre: Skipped 3 previous similar messages [ 9701.304783] Lustre: server umount lustre-OST0000 complete [ 9705.440778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9705.452055] LustreError: Skipped 1 previous similar message [ 9705.460220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9705.490715] Lustre: Skipped 1 previous similar message [ 9712.014230] Lustre: server umount lustre-MDT0000 complete [ 9714.149869] LustreError: 288398:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9714.169551] LustreError: 288398:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9716.843696] LustreError: 291652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790980424 with bad export cookie 1406082792203796703 [ 9716.846240] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9716.850969] LustreError: 291652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9717.535707] Lustre: server umount lustre-MDT0001 complete [ 9726.704901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9731.724957] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9736.789565] Lustre: server umount MGS complete [ 9747.431705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9747.797737] LustreError: 294343:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9747.827187] LustreError: 294356:0:(qmt_pool.c:1648:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9747.903734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9752.560360] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9761.957761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9766.952269] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9776.238335] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9782.916516] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9790.403169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9797.098648] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9804.533865] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9832.932931] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9832.935412] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9832.954074] Lustre: Skipped 1 previous similar message [ 9832.958044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9832.984371] Lustre: Skipped 3 previous similar messages [ 9837.522385] Lustre: server umount lustre-OST0000 complete [ 9847.485550] Lustre: server umount lustre-MDT0000 complete [ 9848.301863] LustreError: 294339:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9848.338311] LustreError: 294339:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9851.513368] LustreError: 294325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790980559 with bad export cookie 1406082792203798642 [ 9851.528766] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9851.532041] LustreError: 294325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9852.019067] Lustre: server umount lustre-MDT0001 complete [ 9865.333938] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 9868.698990] Key type lgssc unregistered [ 9869.199618] LNet: 298290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9869.213132] LNetError: 298290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9869.262421] LNet: Removed LNI 192.168.202.120@tcp [ 9870.013099] Key type .llcrypt unregistered [ 9870.018151] Key type ._llcrypt unregistered [ 9906.883826] Key type ._llcrypt registered [ 9906.886570] Key type .llcrypt registered [ 9906.998297] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 9923.314426] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 9924.924093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9924.986536] alg: No test for adler32 (adler32-zlib) [ 9926.130746] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 9926.370931] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 9928.071205] Key type lgssc registered [ 9929.979739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9964.431716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9964.469477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9965.808134] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9965.860682] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9965.952980] Lustre: lustre-MDT0000: new disk, initializing [ 9966.044587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9966.070207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9970.657756] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9981.837502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9981.984164] Lustre: 302264:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9982.038957] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9982.042972] Lustre: Skipped 1 previous similar message [ 9982.125171] Lustre: lustre-MDT0001: new disk, initializing [ 9982.146982] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9982.256153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9982.287268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9982.290772] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9986.561638] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9996.554248] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10002.956198] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10009.694551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10010.096583] Lustre: lustre-OST0000: new disk, initializing [10010.101288] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10010.106710] Lustre: 304232:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10010.225695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10015.993757] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10016.867135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10016.879705] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10017.087222] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10025.042812] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10032.610761] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [10043.686528] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [10051.107836] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [10058.900816] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [10099.170098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10099.177175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10099.206025] LustreError: Skipped 1 previous similar message [10099.239834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10099.243696] Lustre: Skipped 1 previous similar message [10099.277518] Lustre: Skipped 1 previous similar message [10103.474925] Lustre: server umount lustre-OST0000 complete [10109.412633] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10109.419161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10109.449917] Lustre: Skipped 1 previous similar message [10109.467681] Lustre: Skipped 1 previous similar message [10113.551438] Lustre: server umount lustre-MDT0000 complete [10117.627667] LustreError: 302259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790980825 with bad export cookie 13359537975543969484 [10117.628986] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10117.639895] LustreError: 302259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10118.103604] Lustre: server umount lustre-MDT0001 complete [10126.203555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10131.502786] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10136.616431] Lustre: server umount MGS complete [10149.141839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10149.558549] LustreError: 308149:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10149.579535] LustreError: 308149:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10149.587704] LustreError: 308162:0:(qmt_pool.c:1648:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [10149.638124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10153.867585] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10154.977354] LustreError: 308150:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10159.073424] LustreError: 308149:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10161.850581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10162.259234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10166.340467] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10172.993194] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10178.146981] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10184.544682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10185.086129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10191.892964] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10199.662743] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10230.241163] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10230.251224] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10230.254266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10230.260053] Lustre: Skipped 1 previous similar message [10230.276598] Lustre: Skipped 1 previous similar message [10235.362030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10235.364961] Lustre: Skipped 1 previous similar message [10243.039691] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10243.190731] Lustre: server umount lustre-OST0000 complete [10249.184256] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10249.189818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10249.217914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10249.223409] Lustre: Skipped 2 previous similar messages [10253.678330] Lustre: server umount lustre-MDT0000 complete [10255.840980] LustreError: 308149:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10255.861454] LustreError: 308149:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10257.362476] LustreError: 310872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790980965 with bad export cookie 13359537975543971325 [10257.365037] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10257.371094] LustreError: 310872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10257.768348] Lustre: server umount lustre-MDT0001 complete [10269.334352] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [10272.724494] Key type lgssc unregistered [10273.180502] LNet: 312092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10273.190828] LNetError: 312092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10273.216975] LNet: Removed LNI 192.168.202.120@tcp [10274.322402] Key type .llcrypt unregistered [10274.325419] Key type ._llcrypt unregistered [10297.848926] Key type ._llcrypt registered [10297.853176] Key type .llcrypt registered [10297.968767] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [10313.392980] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [10314.385668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10314.416691] alg: No test for adler32 (adler32-zlib) [10315.565299] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [10315.865745] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [10317.559252] Key type lgssc registered [10318.918642] Lustre: Echo OBD driver; http://www.lustre.org/ [10344.727601] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [10360.941355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10360.970196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10362.290815] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10362.332673] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10362.420729] Lustre: lustre-MDT0000: new disk, initializing [10362.559129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10362.589759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10367.697831] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10379.026934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10379.363711] Lustre: lustre-OST0000: new disk, initializing [10379.366740] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10379.371439] Lustre: Skipped 1 previous similar message [10379.379652] Lustre: 316158:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10379.457168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10385.709722] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [10389.140769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10389.148184] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10389.202704] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11202.705876] Lustre: server umount lustre-MDT0000 complete [11206.648826] LustreError: 315344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790981914 with bad export cookie 8250118473284380978 [11206.667922] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11217.037733] Lustre: server umount lustre-OST0000 complete [11245.719879] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [11255.634728] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [11291.862497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11292.054462] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11292.074386] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11292.181373] Lustre: lustre-MDT0000: new disk, initializing [11292.267420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11292.290910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11297.027438] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11307.984313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11308.116894] Lustre: 320492:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11308.182180] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11308.186559] Lustre: Skipped 1 previous similar message [11308.287666] Lustre: lustre-MDT0001: new disk, initializing [11308.377539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11308.415056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11308.436610] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11312.869534] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11322.686891] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11328.821366] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11335.447852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11335.881682] Lustre: lustre-OST0000: new disk, initializing [11335.891939] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11335.905210] Lustre: 322459:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11336.009928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11342.844841] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11343.442793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11343.454926] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11343.619635] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11353.162491] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11357.857574] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11358.042911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11362.923370] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11363.070514] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11369.443989] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11369.452360] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11369.477095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11374.571279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11374.579644] Lustre: Skipped 2 previous similar messages [11379.167323] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11379.387715] Lustre: server umount lustre-OST0000 complete [11384.801996] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11384.827915] Lustre: Skipped 2 previous similar messages [11384.838783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11384.847525] Lustre: Skipped 1 previous similar message [11389.922354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11389.932572] Lustre: Skipped 1 previous similar message [11395.040809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11395.044343] Lustre: Skipped 1 previous similar message [11398.111163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11398.293707] Lustre: server umount lustre-MDT0000 complete [11402.041877] LustreError: 320484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790982110 with bad export cookie 8250118473287321797 [11402.045769] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11402.061725] LustreError: 320484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11402.600801] Lustre: server umount lustre-MDT0001 complete [11405.892795] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [11407.508338] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 19:01:54 (1790982114) [11413.648341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11414.148023] LustreError: 324786:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11414.154651] LustreError: 324786:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11414.204045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11418.513427] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11419.626912] LustreError: 324786:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11424.739618] LustreError: 324787:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11427.621823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11428.157559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11433.435221] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11442.457566] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11447.945220] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11454.660121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11454.996866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11461.055569] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11468.866307] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11477.108567] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11477.453134] Lustre: lustre-OST0001: new disk, initializing [11477.455629] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11477.473953] Lustre: 327624:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11477.669102] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11482.709867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11482.719736] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11482.758395] LustreError: 327910:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11483.813465] LustreError: 327910:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11483.818772] LustreError: 327910:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [11484.292853] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11484.841907] LustreError: 327918:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11484.847678] LustreError: 327918:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [11486.948278] LustreError: 327918:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11486.952598] LustreError: 327918:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [11491.115656] LustreError: 327918:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11491.137722] LustreError: 327918:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [11493.891331] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11494.334450] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11499.382818] LustreError: 327910:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11499.395106] LustreError: 327910:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 14 previous similar messages [11501.356417] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff95c110e0e800.ost_server_uuid 50 [11503.305515] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff95c110e0e800.ost_server_uuid in FULL state after 0 sec [11508.933539] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff95c110e0e800.ost_server_uuid 50 [11510.307476] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff95c110e0e800.ost_server_uuid in FULL state after 0 sec [11512.790375] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11528.160507] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11528.165155] LustreError: Skipped 1 previous similar message [11528.178668] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11528.186427] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11528.202610] Lustre: Skipped 1 previous similar message [11531.763298] Lustre: server umount lustre-OST0001 complete [11534.308166] LustreError: 329159:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11534.322675] LustreError: 329159:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11539.436622] LustreError: 327920:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11539.438860] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11539.455483] LustreError: 327920:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11539.484139] Lustre: Skipped 1 previous similar message [11543.940547] Lustre: server umount lustre-OST0000 complete [11549.680749] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11549.691461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11549.707058] Lustre: Skipped 1 previous similar message [11549.729196] Lustre: Skipped 7 previous similar messages [11554.788460] LustreError: 324786:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11554.855797] Lustre: server umount lustre-MDT0000 complete [11559.004988] LustreError: 327623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790982267 with bad export cookie 8250118473287322672 [11559.013851] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11559.028772] LustreError: 327623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11559.639797] Lustre: server umount lustre-MDT0001 complete [11571.067890] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [11573.989707] Key type lgssc unregistered [11574.378990] LNet: 330314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11574.388771] LNetError: 330314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11574.406420] LNet: Removed LNI 192.168.202.120@tcp [11575.474344] Key type .llcrypt unregistered [11575.476026] Key type ._llcrypt unregistered [11600.330544] Key type ._llcrypt registered [11600.334876] Key type .llcrypt registered [11600.434975] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [11601.731166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11601.951909] alg: No test for adler32 (adler32-zlib) [11602.922241] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [11603.186074] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [11604.879516] Key type lgssc registered [11606.242886] Lustre: Echo OBD driver; http://www.lustre.org/ [11614.237374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11614.253397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11615.645936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11619.554889] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11625.953870] LustreError: 331749:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11625.985384] LustreError: 331749:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11627.145271] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11631.073531] LustreError: 331750:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11633.804376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11634.096695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11638.697267] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11647.812284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11648.155379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11654.746245] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11659.449383] LustreError: 333429:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11660.535624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [11661.461307] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11665.899644] LustreError: 334044:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11665.939430] LustreError: 334044:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11667.578826] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11667.801395] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11673.078127] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [11673.196662] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11680.417398] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11686.351077] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [11686.578115] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11785.701381] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11785.703019] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11785.712517] LustreError: Skipped 1 previous similar message [11785.713986] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11785.753330] Lustre: Skipped 1 previous similar message [11790.319218] Lustre: server umount lustre-OST0001 complete [11790.837143] LustreError: 334044:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11797.991751] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11797.997926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11798.014268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11798.022487] Lustre: Skipped 1 previous similar message [11799.011684] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11799.027923] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11799.043837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11801.058198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11801.061383] LustreError: 334773:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11801.073016] Lustre: Skipped 1 previous similar message [11801.119425] LustreError: 334773:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11803.859984] Lustre: server umount lustre-OST0000 complete [11808.229071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11808.235893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11808.256154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11813.622681] Lustre: server umount lustre-MDT0000 complete [11816.796024] LustreError: 331730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790982524 with bad export cookie 11311285816590399933 [11816.800696] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11817.103259] Lustre: server umount lustre-MDT0001 complete [11828.079478] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [11830.814857] Key type lgssc unregistered [11831.046882] LNet: 336692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11831.052695] LNetError: 336692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11831.068953] LNet: Removed LNI 192.168.202.120@tcp [11831.773311] Key type .llcrypt unregistered [11831.776309] Key type ._llcrypt unregistered [11874.905227] Key type ._llcrypt registered [11874.912936] Key type .llcrypt registered [11875.038943] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [11875.913852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11875.996543] alg: No test for adler32 (adler32-zlib) [11877.104624] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [11877.376743] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [11878.999289] Key type lgssc registered [11880.381208] Lustre: Echo OBD driver; http://www.lustre.org/ [11889.404804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11889.425092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11890.739982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11895.451757] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11900.901883] LustreError: 338999:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11900.931021] LustreError: 338999:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11905.849557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11906.016936] LustreError: 339000:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11906.470933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11911.549711] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11914.557988] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11921.164530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11921.665661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11928.874115] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11931.877326] LustreError: 340489:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11935.983430] LustreError: 341018:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11935.999790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [11936.011387] LustreError: 341018:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11937.106101] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11937.316672] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11942.410333] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [11944.231613] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [11954.854660] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11962.383865] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11978.224031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11978.227089] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11978.229704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11978.282186] Lustre: Skipped 3 previous similar messages [11981.104685] Lustre: server umount lustre-MDT0000 complete [11983.343425] LustreError: 338995:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11983.362554] LustreError: 338995:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11985.220180] LustreError: 339738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790982693 with bad export cookie 18034939678986327701 [11985.222213] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11985.226153] LustreError: 339738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11985.604430] Lustre: server umount lustre-MDT0001 complete [12000.159507] Lustre: server umount lustre-OST0000 complete [12004.575546] Lustre: 338252:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790982696/real 1790982696] req@ffff97acce9d8380 x1877981355112320/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790982712 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12004.614059] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12004.999282] Lustre: server umount lustre-OST0001 complete [12023.349645] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [12035.754332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12036.094795] LustreError: 344262:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12036.186180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12039.421849] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12046.847547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12047.657272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12053.711415] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12057.143449] Lustre: 345402:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [12063.520772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12063.840235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12069.656337] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12072.069437] LustreError: 345756:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12072.101547] LustreError: 345756:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12078.213223] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12081.878902] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [12081.880963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [12086.003502] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12093.938693] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12097.454942] Lustre: 347273:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12102.631798] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12102.635264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12102.643802] Lustre: Skipped 2 previous similar messages [12102.651910] Lustre: Skipped 5 previous similar messages [12107.755678] LustreError: 344263:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12107.766121] LustreError: 344263:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12107.825570] Lustre: server umount lustre-MDT0000 complete [12111.889942] LustreError: 344243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790982819 with bad export cookie 18034939678986329402 [12111.900328] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12111.907783] LustreError: 344243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12112.320448] Lustre: server umount lustre-MDT0001 complete [12127.792902] Lustre: server umount lustre-OST0000 complete [12128.736196] Lustre: 338251:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790982820/real 1790982820] req@ffff97acce201500 x1877981355160448/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790982836 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12128.786554] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12128.799922] Lustre: Skipped 1 previous similar message [12132.245056] Lustre: server umount lustre-OST0001 complete [12136.933708] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [12139.149796] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [12141.805620] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 19:14:08 (1790982848) [12168.573170] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [12178.605802] loop0: detected capacity change from 0 to 1099511627776 [12179.462177] SGI XFS with ACLs, security attributes, quota, no debug enabled [12179.484699] *** VALIDATE xfs *** [12179.510375] XFS (loop0): Mounting V5 Filesystem [12179.542751] XFS (loop0): Ending clean mount [12181.067220] loop1: detected capacity change from 0 to 18691697672192 [12309.397192] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [12320.902833] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [12365.810615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12366.228496] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12366.288604] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12366.425868] Lustre: lustre-MDT0000: new disk, initializing [12366.553205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12366.565857] Lustre: Skipped 1 previous similar message [12366.603786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12372.444298] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12382.773195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12382.885910] Lustre: 352416:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12382.946323] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12382.952348] Lustre: Skipped 1 previous similar message [12383.010279] Lustre: lustre-MDT0001: new disk, initializing [12383.080501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12383.093109] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12387.739822] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12398.610661] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12406.453684] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12413.513570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12413.755131] Lustre: lustre-OST0000: new disk, initializing [12413.758740] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12413.765935] Lustre: 354385:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12415.819817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12415.835232] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12416.008109] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12420.418455] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12430.332259] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12435.123756] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12435.327878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12440.813512] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12441.065223] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12446.694822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12446.698266] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12446.718586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12446.729628] Lustre: Skipped 1 previous similar message [12446.758420] LustreError: Skipped 1 previous similar message [12449.283578] Lustre: server umount lustre-OST0000 complete [12454.883556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12454.893895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12454.908268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12454.911022] Lustre: Skipped 1 previous similar message [12460.054218] Lustre: server umount lustre-MDT0000 complete [12462.050332] LustreError: 354155:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12462.071839] LustreError: 354155:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12464.345493] LustreError: 352409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790983172 with bad export cookie 18034939678986330956 [12464.362340] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12464.364428] LustreError: 352409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12464.688741] Lustre: server umount lustre-MDT0001 complete [12473.454148] XFS (loop0): Unmounting Filesystem [12478.336596] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 19:19:45 (1790983185) [12485.295575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12485.984767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12485.989119] Lustre: Skipped 2 previous similar messages [12490.151692] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12498.344697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12503.575242] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12511.464819] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12517.202994] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12523.341239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12530.520423] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12539.254121] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12549.625925] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12549.637077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12549.659780] Lustre: Skipped 1 previous similar message [12549.670699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12549.679229] Lustre: Skipped 2 previous similar messages [12554.555196] Lustre: server umount lustre-OST0000 complete [12560.354126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12560.373162] LustreError: Skipped 1 previous similar message [12560.395114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12560.403094] Lustre: Skipped 2 previous similar messages [12565.528833] Lustre: server umount lustre-MDT0000 complete [12570.259158] LustreError: 356924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790983278 with bad export cookie 18034939678986331838 [12570.266300] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12570.270244] LustreError: 356924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12570.776594] Lustre: server umount lustre-MDT0001 complete [12584.351413] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [12587.762211] Key type lgssc unregistered [12588.126986] LNet: 360570:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12588.131222] LNetError: 360570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12589.170295] LNet: Removed LNI 192.168.202.120@tcp [12590.500509] Key type .llcrypt unregistered [12590.502249] Key type ._llcrypt unregistered [12620.935250] Key type ._llcrypt registered [12620.937624] Key type .llcrypt registered [12621.010275] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [12622.705353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12622.799894] alg: No test for adler32 (adler32-zlib) [12623.991985] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [12624.387711] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [12626.047194] Key type lgssc registered [12627.674845] Lustre: Echo OBD driver; http://www.lustre.org/ [12636.577300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12636.602765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12638.073335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12643.555462] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12648.420212] LustreError: 362156:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12648.458845] LustreError: 362156:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12653.445768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12653.536639] LustreError: 362157:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12654.032420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12658.831184] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12666.791278] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12672.206844] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12679.292874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12679.733476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12686.691821] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12694.581094] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12705.264483] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12705.283925] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12705.319264] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12706.713732] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.20@tcp (stopping) [12707.317345] Lustre: server umount lustre-MDT0001 complete [12708.327266] LustreError: 362156:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12708.328628] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12711.815766] LustreError: 362151:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12711.838220] LustreError: 362151:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12716.950712] LustreError: 362152:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12716.970770] LustreError: 362152:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12718.000424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12718.167865] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12718.174436] Lustre: Found index 1 for lustre-MDT0001, updating log [12718.228582] Lustre: 362146:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12718.521719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12723.160213] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [12723.705106] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12726.809602] LustreError: 365755:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12726.831477] LustreError: 365755:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12726.850542] LustreError: 362148:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12726.953232] LustreError: 365758:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.120@tcp: cfg command failed: rc = -17 [12726.963850] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [12726.963850] [13029.274100] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.20@tcp (stopping) [13030.881616] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13030.886710] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13034.388191] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.20@tcp (stopping) [13034.396980] Lustre: Skipped 2 previous similar messages [13035.341940] Lustre: server umount lustre-MDT0001 complete [13036.009814] LustreError: 367888:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13036.028700] LustreError: 367888:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13046.602060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13046.671194] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13046.676592] Lustre: Found index 1 for lustre-MDT0001, updating log [13046.713617] Lustre: 362145:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13047.093445] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13052.288215] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [13052.410879] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13053.954745] LustreError: 374103:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13053.965460] LustreError: 374103:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13053.994058] LustreError: 362148:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13054.006179] LustreError: 362148:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13054.127604] LustreError: 374107:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [13054.138729] LustreError: 374107:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [13054.147057] LustreError: 374107:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.120@tcp: cfg command failed: rc = -17 [13054.172664] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.202.120@tcp [13054.172664] [13359.595733] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13359.612835] Lustre: Skipped 1 previous similar message [13359.626962] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13359.629659] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13359.663601] Lustre: Skipped 1 previous similar message [13365.040819] Lustre: server umount lustre-MDT0001 complete [13367.185177] LustreError: 362153:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13367.211074] LustreError: 362153:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [13376.854187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13377.018675] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13377.026907] Lustre: Found index 1 for lustre-MDT0001, updating log [13377.077561] Lustre: 362145:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13377.537204] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13382.638928] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13382.821935] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [13384.196886] LustreError: 382361:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13384.204421] LustreError: 382361:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13384.228423] LustreError: 362148:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13384.238690] LustreError: 362148:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13384.324344] LustreError: 382371:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [13384.339980] LustreError: 382371:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.120@tcp: cfg command failed: rc = -114 [13384.346275] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [13384.346275] [13710.316029] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13710.322177] Lustre: Skipped 1 previous similar message [13710.322580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13710.333143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13710.338413] Lustre: Skipped 5 previous similar messages [13712.148381] LustreError: 389724:0:(ldlm_resource.c:1227:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000abeadf41) refcount nonzero (2) after lock cleanup; forcing cleanup. [13712.294643] Lustre: server umount lustre-MDT0000 complete [13715.435425] LustreError: 363320:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13715.459121] LustreError: 363320:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [13716.325462] LustreError: 362138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790984424 with bad export cookie 15906270361593105102 [13716.333451] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13716.335956] LustreError: 362138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13720.545562] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13720.555359] Lustre: Skipped 2 previous similar messages [13722.866164] Lustre: server umount lustre-MDT0001 complete [13728.719430] Lustre: server umount lustre-OST0000 complete [13741.364441] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [13757.671446] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [13798.441857] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 19:41:45 (1790984505) [13806.836539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13807.227164] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13807.301107] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13807.534501] Lustre: lustre-MDT0000: new disk, initializing [13807.627516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13807.651168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13813.540192] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [13824.874851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13825.061608] Lustre: 392710:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13825.152297] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13825.156605] Lustre: Skipped 1 previous similar message [13825.264474] Lustre: lustre-MDT0001: new disk, initializing [13825.400624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13825.453277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13825.476178] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13831.782852] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [13841.923745] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13848.731289] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13856.164184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13856.429822] Lustre: lustre-OST0000: new disk, initializing [13856.432595] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13856.440128] Lustre: 394676:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13856.536758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13861.836626] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [13865.540993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13865.549881] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13865.761547] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13872.684934] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13886.448874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13886.454778] Lustre: Skipped 1 previous similar message [13886.468489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13886.476986] Lustre: Skipped 2 previous similar messages [13888.428554] Lustre: server umount lustre-MDT0000 complete [13891.556743] LustreError: 393788:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13891.576778] LustreError: 393788:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [13892.661691] LustreError: 392701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790984600 with bad export cookie 15906270361593107048 [13892.672604] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13892.952917] Lustre: server umount lustre-MDT0001 complete [13909.311023] Lustre: server umount lustre-OST0000 complete [13940.665716] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [13951.441833] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [13986.983668] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13674 sec ========= 19:44:53 (1790984693) [13988.939117] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:44:55 (1790984695) === [13992.806609] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:44:59 (1790984699) === [14038.534469] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [14041.610889] Key type lgssc unregistered [14041.986755] LNet: 401044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14041.996964] LNetError: 401044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14043.074446] LNet: Removed LNI 192.168.202.120@tcp [14044.228900] Key type .llcrypt unregistered [14044.232058] Key type ._llcrypt unregistered