[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 439158986 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524576K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002158] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004008] kvm-guest: setup PV IPIs [ 0.005895] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006010] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007006] pid_max: default: 32768 minimum: 301 [ 0.008095] LSM: Security Framework initializing [ 0.009024] Yama: becoming mindful. [ 0.010024] SELinux: Initializing. [ 0.010744] *** VALIDATE selinux *** [ 0.016541] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.020003] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.021082] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.023030] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024059] *** VALIDATE tmpfs *** [ 0.025343] *** VALIDATE proc *** [ 0.026157] *** VALIDATE cgroup *** [ 0.027015] *** VALIDATE cgroup2 *** [ 0.027776] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.028087] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.029003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.030017] Spectre V2 : User space: Vulnerable [ 0.031003] Speculative Store Bypass: Vulnerable [ 0.034076] debug: unmapping init [mem 0xffffffff97e59000-0xffffffff97e60fff] [ 0.036205] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.037557] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.038011] ... version: 2 [ 0.039007] ... bit width: 48 [ 0.040006] ... generic registers: 4 [ 0.041006] ... value mask: 0000ffffffffffff [ 0.042006] ... max period: 00007fffffffffff [ 0.043007] ... fixed-purpose events: 3 [ 0.044006] ... event mask: 000000070000000f [ 0.046198] rcu: Hierarchical SRCU implementation. [ 0.048373] smp: Bringing up secondary CPUs ... [ 0.049471] x86: Booting SMP configuration: [ 0.050012] .... node #0, CPUs: #1 #2 #3 [ 0.053017] smp: Brought up 1 node, 4 CPUs [ 0.055008] smpboot: Max logical packages: 1 [ 0.056011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.213810] node 0 deferred pages initialised in 157ms [ 0.216335] devtmpfs: initialized [ 0.217239] x86/mm: Memory block size: 128MB [ 0.219494] gcov: version magic: 0x41383552 [ 0.221204] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.224066] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.226277] pinctrl core: initialized pinctrl subsystem [ 0.227105] [ 0.227539] ************************************************************* [ 0.229007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.231007] ** ** [ 0.233008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.234005] ** ** [ 0.236007] ** This means that this kernel is built to expose internal ** [ 0.238008] ** IOMMU data structures, which may compromise security on ** [ 0.239006] ** your system. ** [ 0.241007] ** ** [ 0.242006] ** If you see this message and you are not debugging the ** [ 0.244007] ** kernel, report this immediately to your vendor! ** [ 0.246008] ** ** [ 0.247007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.249006] ************************************************************* [ 0.251589] NET: Registered protocol family 16 [ 0.252406] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.254038] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.256038] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.259094] cpuidle: using governor menu [ 0.260390] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.261314] PCI: Using configuration type 1 for base access [ 0.263114] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.272095] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.273013] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.275069] cryptd: max_cpu_qlen set to 1000 [ 0.276245] ACPI: Added _OSI(Module Device) [ 0.277000] ACPI: Added _OSI(Processor Device) [ 0.279011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.280008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.283103] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.288379] ACPI: Interpreter enabled [ 0.289034] ACPI: PM: (supports S0 S3 S4 S5) [ 0.290006] ACPI: Using IOAPIC for interrupt routing [ 0.291055] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.294234] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.301303] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.302021] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.304010] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.306048] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.309971] acpiphp: Slot [2] registered [ 0.310066] acpiphp: Slot [5] registered [ 0.311007] acpiphp: Slot [6] registered [ 0.311877] acpiphp: Slot [7] registered [ 0.313046] acpiphp: Slot [8] registered [ 0.314060] acpiphp: Slot [9] registered [ 0.315022] acpiphp: Slot [10] registered [ 0.315964] acpiphp: Slot [3] registered [ 0.317082] acpiphp: Slot [4] registered [ 0.318063] acpiphp: Slot [11] registered [ 0.319105] acpiphp: Slot [12] registered [ 0.320068] acpiphp: Slot [13] registered [ 0.321056] acpiphp: Slot [14] registered [ 0.322045] acpiphp: Slot [15] registered [ 0.323020] acpiphp: Slot [16] registered [ 0.324043] acpiphp: Slot [17] registered [ 0.325042] acpiphp: Slot [18] registered [ 0.326044] acpiphp: Slot [19] registered [ 0.327075] acpiphp: Slot [20] registered [ 0.329076] acpiphp: Slot [21] registered [ 0.330054] acpiphp: Slot [22] registered [ 0.331081] acpiphp: Slot [23] registered [ 0.332063] acpiphp: Slot [24] registered [ 0.333046] acpiphp: Slot [25] registered [ 0.334081] acpiphp: Slot [26] registered [ 0.335048] acpiphp: Slot [27] registered [ 0.336055] acpiphp: Slot [28] registered [ 0.338048] acpiphp: Slot [29] registered [ 0.339045] acpiphp: Slot [30] registered [ 0.340116] acpiphp: Slot [31] registered [ 0.340991] PCI host bridge to bus 0000:00 [ 0.342012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.344008] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.345008] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.347008] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.348010] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.351010] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.352164] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.355189] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.357795] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.365883] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.369934] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.371015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.373008] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.374007] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.376372] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.378537] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.380021] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.382768] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.386007] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.395909] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.399009] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.403025] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.408008] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.412009] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.421015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.430586] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.440021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.445011] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.456013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.464524] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.470007] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.475009] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.494021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.501770] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.505007] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.510011] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.520012] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.527662] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.530892] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.533007] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.542011] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.550011] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.558007] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.562007] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.576026] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.589462] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.591255] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.593242] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.595258] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.596126] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.600229] iommu: Default domain type: Passthrough [ 0.602211] SCSI subsystem initialized [ 0.603085] ACPI: bus type USB registered [ 0.604081] usbcore: registered new interface driver usbfs [ 0.606037] usbcore: registered new interface driver hub [ 0.608047] usbcore: registered new device driver usb [ 0.609125] pps_core: LinuxPPS API ver. 1 registered [ 0.610007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.613024] PTP clock support registered [ 0.614126] EDAC MC: Ver: 3.0.0 [ 0.615374] PCI: Using ACPI for IRQ routing [ 0.616789] NetLabel: Initializing [ 0.617008] NetLabel: domain hash size = 128 [ 0.618006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.619076] NetLabel: unlabeled traffic allowed by default [ 0.620088] vgaarb: loaded [ 0.621179] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.622006] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.626220] clocksource: Switched to clocksource kvm-clock [ 0.721169] VFS: Disk quotas dquot_6.6.0 [ 0.722339] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.724032] *** VALIDATE ramfs *** [ 0.724886] *** VALIDATE hugetlbfs *** [ 0.725940] pnp: PnP ACPI init [ 0.727694] pnp: PnP ACPI: found 6 devices [ 0.739968] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.741827] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.743186] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.744551] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.745953] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.747366] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.748928] NET: Registered protocol family 2 [ 0.750858] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.753880] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.756264] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.760685] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.763364] TCP: Hash tables configured (established 65536 bind 65536) [ 0.765579] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.767972] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.770139] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.772565] NET: Registered protocol family 1 [ 0.774285] RPC: Registered named UNIX socket transport module. [ 0.775783] RPC: Registered udp transport module. [ 0.776722] RPC: Registered tcp transport module. [ 0.777632] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.779617] NET: Registered protocol family 44 [ 0.781030] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.783176] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.785402] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.787523] PCI: CLS 0 bytes, default 64 [ 0.789546] Unpacking initramfs... [ 2.143552] debug: unmapping init [mem 0xffff96087cc54000-0xffff96087ffbffff] [ 2.147172] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.148908] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.151119] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.566781] Initialise system trusted keyrings [ 2.567837] Key type blacklist registered [ 2.569046] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.576130] zbud: loaded [ 2.578928] *** VALIDATE nfs *** [ 2.579817] *** VALIDATE nfs4 *** [ 2.580945] pstore: using deflate compression [ 2.583094] Platform Keyring initialized [ 2.682566] NET: Registered protocol family 38 [ 2.683976] Key type asymmetric registered [ 2.685012] Asymmetric key parser 'x509' registered [ 2.686876] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.689573] io scheduler mq-deadline registered [ 2.690705] io scheduler kyber registered [ 2.691828] io scheduler bfq registered [ 2.694166] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.696070] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.697736] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.700381] ACPI: Power Button [PWRF] [ 2.706834] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.713793] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.735462] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.746924] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.765249] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.791071] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.818256] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.822236] Non-volatile memory driver v1.3 [ 2.823917] Linux agpgart interface v0.103 [ 2.849161] virtio_blk virtio1: [vda] 134584 512-byte logical blocks (68.9 MB/65.7 MiB) [ 2.853242] vda: detected capacity change from 0 to 68907008 [ 2.864590] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.866800] vdb: detected capacity change from 0 to 1073741824 [ 2.882221] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.885253] vdc: detected capacity change from 0 to 2621440000 [ 2.901096] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.902779] vdd: detected capacity change from 0 to 2621440000 [ 2.912980] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.915295] vde: detected capacity change from 0 to 4294967296 [ 2.925945] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.928192] vdf: detected capacity change from 0 to 4294967296 [ 2.932632] libphy: Fixed MDIO Bus: probed [ 2.936375] usbcore: registered new interface driver usbserial_generic [ 2.938139] usbserial: USB Serial support registered for generic [ 2.939802] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.942936] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.944288] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.945967] mousedev: PS/2 mouse device common for all mice [ 2.947902] rtc_cmos 00:05: RTC can wake from S4 [ 2.949676] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.951863] rtc_cmos 00:05: registered as rtc0 [ 2.953687] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.953774] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.956234] intel_pstate: CPU model not supported [ 2.959230] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.962881] hid: raw HID events driver (C) Jiri Kosina [ 2.964166] usbcore: registered new interface driver usbhid [ 2.965581] usbhid: USB HID core driver [ 2.966519] drop_monitor: Initializing network drop monitor service [ 2.967994] Initializing XFRM netlink socket [ 2.969189] NET: Registered protocol family 10 [ 2.971130] Segment Routing with IPv6 [ 2.971819] NET: Registered protocol family 17 [ 2.972895] mpls_gso: MPLS GSO support [ 2.978096] RAS: Correctable Errors collector initialized. [ 2.979524] AVX version of gcm_enc/dec engaged. [ 2.980660] AES CTR mode by8 optimization enabled [ 3.019632] sched_clock: Marking stable (3019613257, 0)->(3742128793, -722515536) [ 3.022663] registered taskstats version 1 [ 3.023954] Loading compiled-in X.509 certificates [ 3.025779] zswap: loaded using pool lzo/zbud [ 3.043163] Key type big_key registered [ 3.051876] Key type encrypted registered [ 3.053553] ima: No TPM chip found, activating TPM-bypass! [ 3.054810] ima: Allocated hash algorithm: sha1 [ 3.056026] ima: No architecture policies found [ 3.057065] evm: Initialising EVM extended attributes: [ 3.058374] evm: security.selinux [ 3.059288] evm: security.ima [ 3.059936] evm: security.capability [ 3.060914] evm: HMAC attrs: 0x1 [ 3.062344] rtc_cmos 00:05: setting system clock to 2026-03-16 13:34:37 UTC (1773668077) [ 3.065872] debug: unmapping init [mem 0xffffffff98e03000-0xffffffff98ffffff] [ 3.067825] debug: unmapping init [mem 0xffffffff97b82000-0xffffffff97e58fff] [ 3.075054] Write protecting the kernel read-only data: 28672k [ 3.077234] debug: unmapping init [mem 0xffffffff96203000-0xffffffff963fffff] [ 3.078955] debug: unmapping init [mem 0xffffffff96b14000-0xffffffff96bfffff] [ 3.102835] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.108082] systemd[1]: Detected virtualization kvm. [ 3.109125] systemd[1]: Detected architecture x86-64. [ 3.110151] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.140783] systemd[1]: No hostname configured. [ 3.141975] systemd[1]: Set hostname to . [ 3.143406] random: systemd: uninitialized urandom read (16 bytes read) [ 3.145084] systemd[1]: Initializing machine ID from random generator. [ 3.174803] random: ln: uninitialized urandom read (6 bytes read) [ 3.288612] random: systemd: uninitialized urandom read (16 bytes read) [ 3.292030] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.295912] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.305253] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.818362] device-mapper: uevent: version 1.0.3 [ 3.819561] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.415065] random: fast init done [ 4.419448] scsi host0: ata_piix [ 4.449553] scsi host1: ata_piix [ 4.461457] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.463743] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 4.499122] virtio_net virtio0 ens2: renamed from eth0 [ 8.124163] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.567555] random: crng init done [ 9.569247] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.856400] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.852429] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.033777] SELinux: Disabled at runtime. [ 11.087306] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.101826] systemd[1]: Detected virtualization kvm. [ 11.103392] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.462424] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.464850] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.467620] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.471265] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.473758] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.478670] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.483548] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd File Systems. [ 11.532153] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 11.797155] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.106743] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.112407] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.199917] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.211348] EDAC sbridge: Ver: 1.1.2 [ 13.203493] Key type dns_resolver registered [ 13.445875] NFS: Registering the id_resolver key type [ 13.446891] Key type id_resolver registered [ 13.448013] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg328-server login: [ 34.758450] libcfs: loading out-of-tree module taints kernel. [ 34.775939] Key type ._llcrypt registered [ 34.776675] Key type .llcrypt registered [ 34.818422] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 43.056802] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 43.765249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 43.774114] alg: No test for adler32 (adler32-zlib) [ 44.909843] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 45.383307] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 47.064145] Key type lgssc registered [ 48.120818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 56.865510] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 75.195375] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 81.329038] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 81.341649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 82.464765] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 82.485368] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 82.528679] Lustre: lustre-MDT0000: new disk, initializing [ 82.560976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 82.569525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 84.035748] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 89.568043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 89.610862] Lustre: 6487:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 89.625253] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 89.629262] Lustre: Skipped 1 previous similar message [ 89.673820] Lustre: lustre-MDT0001: new disk, initializing [ 89.707309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 89.717429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 89.722557] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 91.127438] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 93.380696] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 96.674543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 96.766357] Lustre: lustre-OST0000: new disk, initializing [ 96.768618] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 96.784664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 98.612196] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 98.923574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 98.926431] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 98.944048] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 103.951556] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 103.994135] Lustre: lustre-OST0001: new disk, initializing [ 103.996331] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 104.012076] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 105.828510] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 106.348612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 106.351914] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 106.372302] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 111.769561] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 119.423931] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 126.317763] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing check_logdir /tmp/testlogs/ [ 127.739024] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing yml_node [ 129.162449] Lustre: DEBUG MARKER: Client: 2.17.51.1 [ 129.978528] Lustre: DEBUG MARKER: MDS: 2.17.51.1 [ 130.830868] Lustre: DEBUG MARKER: OSS: 2.17.51.1 [ 131.399097] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Mar 16 09:36:44 EDT 2026 [ 137.247029] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 137.696856] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 151.519452] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 151.521549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 151.524745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 152.543564] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 152.544082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 152.549547] Lustre: Skipped 1 previous similar message [ 152.552339] Lustre: Skipped 1 previous similar message [ 155.481239] Lustre: server umount lustre-MDT0000 complete [ 156.728170] LustreError: 6478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773668231 with bad export cookie 4848154399402291786 [ 156.729465] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 156.732507] LustreError: 6478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 157.663585] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 157.663925] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 157.667038] Lustre: Skipped 2 previous similar messages [ 157.670326] Lustre: Skipped 1 previous similar message [ 161.695433] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 161.697248] Lustre: Skipped 1 previous similar message [ 162.907921] Lustre: server umount lustre-MDT0001 complete [ 164.206164] Lustre: server umount lustre-OST0000 complete [ 165.424484] Lustre: server umount lustre-OST0001 complete [ 167.300920] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 170.140674] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 182.055374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.150371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 182.162457] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 182.202136] Lustre: lustre-MDT0000: new disk, initializing [ 182.229285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 182.236857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 183.543975] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 188.020713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 188.067067] Lustre: 14950:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 188.201435] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 189.563627] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 192.902412] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 194.379768] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 196.827133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 196.929908] Lustre: lustre-OST0000: new disk, initializing [ 196.931793] Lustre: Skipped 1 previous similar message [ 196.934218] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 196.940167] Lustre: Skipped 2 previous similar messages [ 196.973757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.976228] Lustre: Skipped 1 previous similar message [ 198.908081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 198.911141] Lustre: Skipped 1 previous similar message [ 198.912707] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 198.958731] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 198.982743] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 202.235889] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 203.687730] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 203.765683] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 205.151050] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 205.231673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 207.957941] Lustre: server umount lustre-OST0000 complete [ 209.376411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 209.377277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 209.380857] Lustre: Skipped 1 previous similar message [ 218.617783] Lustre: server umount lustre-MDT0000 complete [ 219.616729] LustreError: 14963:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 219.623458] LustreError: 14963:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 6 previous similar messages [ 219.878107] LustreError: 14942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773668294 with bad export cookie 4848154399402293305 [ 219.882326] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 219.882651] LustreError: 14942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 230.416363] Lustre: server umount lustre-MDT0001 complete [ 232.221555] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 09:38:25 (1773668305) [ 234.550418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 234.720995] LustreError: 18811:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 234.751726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 236.100972] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 239.044306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 240.501731] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 242.379178] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 243.572243] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 245.955435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 248.013764] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 249.926606] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 267.412597] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 270.820434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 270.822432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 270.826222] Lustre: Skipped 2 previous similar messages [ 270.830198] Lustre: Skipped 5 previous similar messages [ 276.633698] Lustre: server umount lustre-MDT0000 complete [ 277.883142] LustreError: 18792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773668352 with bad export cookie 4848154399402294187 [ 277.886282] LustreError: 18792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 277.886393] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 278.007338] Lustre: server umount lustre-MDT0001 complete [ 289.568740] Lustre: server umount lustre-OST0000 complete [ 297.036282] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 301.056493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 301.225501] LustreError: 22857:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 301.244444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 301.248048] Lustre: Skipped 2 previous similar messages [ 302.740524] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 306.436594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 308.196112] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 309.435232] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 312.089430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 314.467984] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 317.525691] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 317.592755] Lustre: lustre-OST0001: new disk, initializing [ 317.594645] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 319.554784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 319.558294] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 319.591858] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 319.650945] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 324.347987] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 325.880895] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 337.377656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 337.380227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 337.385690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 337.387505] Lustre: Skipped 3 previous similar messages [ 338.728651] Lustre: server umount lustre-MDT0000 complete [ 340.135306] LustreError: 22839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773668414 with bad export cookie 4848154399402295034 [ 340.139770] LustreError: 22839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 340.141096] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 341.473838] LustreError: 26582:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 341.481071] LustreError: 26582:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 342.260764] Lustre: server umount lustre-MDT0001 complete [ 343.754728] Lustre: server umount lustre-OST0000 complete [ 345.147269] Lustre: server umount lustre-OST0001 complete [ 351.118156] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 354.821378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 354.984240] LustreError: 28154:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 356.422983] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 359.486050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 361.074961] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 362.090764] Lustre: 29269:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 364.513279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 365.670022] LustreError: 29621:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 365.680477] LustreError: 29621:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 366.621359] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 369.670803] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 369.761765] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 369.764569] Lustre: Skipped 6 previous similar messages [ 371.748062] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 374.970760] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 376.216119] Lustre: 31119:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 385.507223] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 385.508840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 385.511330] Lustre: Skipped 6 previous similar messages [ 385.513501] Lustre: Skipped 3 previous similar messages [ 385.515905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 389.143873] Lustre: server umount lustre-MDT0000 complete [ 390.382525] LustreError: 28137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773668464 with bad export cookie 4848154399402296329 [ 390.385482] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 390.387394] LustreError: 28137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 390.489019] Lustre: server umount lustre-MDT0001 complete [ 401.986021] Lustre: server umount lustre-OST0000 complete [ 413.466252] Lustre: server umount lustre-OST0001 complete [ 416.446661] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 09:41:29 (1773668489) [ 425.564043] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 429.413763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 429.579414] LustreError: 33900:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 431.086365] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 434.640318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 436.337345] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 437.480601] Lustre: 35013:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 440.047965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 442.412951] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 445.442351] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 447.544461] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 461.386787] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 462.797053] Lustre: 36885:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 470.499887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 470.504713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 470.513350] Lustre: Skipped 1 previous similar message [ 470.514400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 475.703504] Lustre: server umount lustre-MDT0000 complete [ 479.731638] LustreError: 33880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773668554 with bad export cookie 4848154399402297435 [ 479.746218] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 479.999688] Lustre: server umount lustre-MDT0001 complete [ 493.924549] Lustre: server umount lustre-OST0000 complete [ 496.928169] Lustre: 3643:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773668555/real 1773668555] req@ffff9608ff9ef480 x1859825821815552/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773668571 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 497.128847] Lustre: server umount lustre-OST0001 complete [ 504.361775] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 09:42:56 (1773668576) [ 517.367059] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 521.856130] hrtimer: interrupt took 2187509 ns [ 529.056779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 529.576952] LustreError: 39010:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 529.594984] LustreError: 39010:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 529.797221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 529.815300] Lustre: Skipped 4 previous similar messages [ 536.545569] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 551.054176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 556.527114] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 558.919830] Lustre: 40125:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 564.026701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 568.921275] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 575.604027] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 579.948517] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 585.762503] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 593.735425] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 593.738721] Lustre: Skipped 1 previous similar message [ 596.454776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 596.463727] Lustre: Skipped 6 previous similar messages [ 601.570622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 601.578094] Lustre: Skipped 16 previous similar messages [ 602.360142] Lustre: server umount lustre-MDT0000 complete [ 602.396075] LustreError: 39007:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 602.403248] LustreError: 39007:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 6 previous similar messages [ 604.855562] LustreError: 38991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773668679 with bad export cookie 4848154399402298541 [ 604.859966] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 604.862411] LustreError: 38991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 611.188547] Lustre: server umount lustre-MDT0001 complete [ 617.874387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 630.239805] LustreError: 3642:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9607c8607b80 x1859825821866496/t0(0) o250->MGC192.168.203.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 634.243460] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 639.852587] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 641.683952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 645.497389] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 647.670145] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 647.674166] Lustre: Skipped 1 previous similar message [ 649.588424] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 653.691320] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 656.192753] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 657.175472] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 662.498798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 666.257233] Lustre: server umount lustre-MDT0000 complete [ 668.989328] LustreError: 38992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773668743 with bad export cookie 4848154399402299731 [ 668.996629] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 668.998806] LustreError: 38992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 669.245334] Lustre: server umount lustre-MDT0001 complete [ 682.556256] Lustre: server umount lustre-OST0000 complete [ 694.892437] Lustre: server umount lustre-OST0001 complete [ 701.296637] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 09:46:14 (1773668774) [ 713.699144] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 721.299762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 725.196480] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 731.358941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 733.830400] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 735.433654] Lustre: 47467:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 735.439176] Lustre: 47467:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 739.061020] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 741.356154] LustreError: 48057:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 741.371505] LustreError: 48057:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 18 previous similar messages [ 742.112045] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 746.602849] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 749.510849] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 754.235644] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 762.091407] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 772.577673] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 772.587240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 772.598806] Lustre: Skipped 7 previous similar messages [ 776.114538] Lustre: server umount lustre-MDT0000 complete [ 778.909557] LustreError: 46333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773668853 with bad export cookie 4848154399402300529 [ 778.919359] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 779.183928] Lustre: server umount lustre-MDT0001 complete [ 792.152349] Lustre: server umount lustre-OST0000 complete [ 805.208780] Lustre: server umount lustre-OST0001 complete [ 810.972863] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 09:48:04 (1773668884) [ 812.505663] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 813.558622] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 09:48:06 (1773668886) [ 820.276790] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 825.466253] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 839.890448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 840.122588] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 840.190342] Lustre: lustre-MDT0000: new disk, initializing [ 840.254441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 840.259800] Lustre: Skipped 9 previous similar messages [ 840.276531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 842.934830] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 848.182158] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 851.918903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 853.173582] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 853.195810] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 855.483190] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 859.925594] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 861.480609] Lustre: DEBUG MARKER: create test files [ 884.197307] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 884.206375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 884.213936] Lustre: Skipped 15 previous similar messages [ 887.382968] Lustre: server umount lustre-OST0000 complete [ 889.929217] Lustre: server umount lustre-MDT0000 complete [ 896.276849] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 898.018649] Key type lgssc unregistered [ 898.197885] LNet: 54865:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.203789] LNetError: 54865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 898.219734] LNet: Removed LNI 192.168.203.128@tcp [ 898.759148] Key type .llcrypt unregistered [ 898.762858] Key type ._llcrypt unregistered [ 916.955084] Key type ._llcrypt registered [ 916.956615] Key type .llcrypt registered [ 917.020600] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 917.721641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 917.733112] alg: No test for adler32 (adler32-zlib) [ 918.628354] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 918.733267] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 920.343152] Key type lgssc registered [ 920.862804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 925.994990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 926.021251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 927.350059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 929.991409] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 933.448358] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 937.827794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 938.074100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 942.030674] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 945.364099] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 952.358342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 952.407786] Lustre: 57447:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 510 < left 544, rollback = 9 [ 952.410722] Lustre: 57447:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/5, destroy: 0/0/0 [ 952.412965] Lustre: 57447:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 952.415192] Lustre: 57447:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 952.417708] Lustre: 57447:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 952.420740] Lustre: 57447:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 959.734978] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 962.935573] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 967.656381] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 967.662639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 967.671620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 972.285669] Lustre: server umount lustre-OST0000 complete [ 974.468499] Lustre: server umount lustre-MDT0000 complete [ 980.057199] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 981.557229] Key type lgssc unregistered [ 981.722367] LNet: 58950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.727040] LNetError: 58950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 981.741983] LNet: Removed LNI 192.168.203.128@tcp [ 982.224797] Key type .llcrypt unregistered [ 982.226685] Key type ._llcrypt unregistered [ 998.283147] Key type ._llcrypt registered [ 998.284739] Key type .llcrypt registered [ 998.346681] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 998.933620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 998.947272] alg: No test for adler32 (adler32-zlib) [ 999.902589] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 1000.072907] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1001.663132] Key type lgssc registered [ 1002.196313] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1006.538965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1006.552361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1007.810587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1009.967675] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1013.125353] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1016.809788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1016.992145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1018.024295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1018.066364] Lustre: 61737:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 510 < left 544, rollback = 9 [ 1018.071603] Lustre: 61737:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/5, destroy: 0/0/0 [ 1018.077517] Lustre: 61737:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1018.080835] Lustre: 61737:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 1018.085354] Lustre: 61737:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1018.089694] Lustre: 61737:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1020.193750] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1022.933989] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1026.471688] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1033.700820] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1033.703760] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1033.708498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1036.936277] Lustre: server umount lustre-OST0000 complete [ 1039.422434] Lustre: server umount lustre-MDT0000 complete [ 1046.258913] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1047.986569] Key type lgssc unregistered [ 1048.162477] LNet: 63061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1048.168171] LNetError: 63061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1048.179103] LNet: Removed LNI 192.168.203.128@tcp [ 1048.689158] Key type .llcrypt unregistered [ 1048.691300] Key type ._llcrypt unregistered [ 1059.404081] Key type ._llcrypt registered [ 1059.406346] Key type .llcrypt registered [ 1059.482075] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 1069.007931] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1069.528730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1069.608090] alg: No test for adler32 (adler32-zlib) [ 1070.565546] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 1070.731100] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1072.383179] Key type lgssc registered [ 1073.190615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1095.671520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1095.689351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1096.829835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1096.852618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1096.906949] Lustre: lustre-MDT0000: new disk, initializing [ 1096.973816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1096.993949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1099.294836] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1107.466594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1107.761067] Lustre: 66158:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1107.824606] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1107.830904] Lustre: Skipped 1 previous similar message [ 1107.931988] Lustre: lustre-MDT0001: new disk, initializing [ 1108.134232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1108.224931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1108.240823] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1114.285091] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1123.332187] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1127.381625] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1135.214347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1135.631200] Lustre: lustre-OST0000: new disk, initializing [ 1135.641266] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1135.753256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1142.916458] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1143.361322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1143.371766] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1143.565569] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1152.558218] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1158.340579] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1158.712354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1163.571160] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1163.943336] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1169.393564] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1169.402607] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1169.422177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1169.437466] Lustre: Skipped 1 previous similar message [ 1169.464442] Lustre: Skipped 1 previous similar message [ 1172.299207] Lustre: server umount lustre-OST0000 complete [ 1179.620125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1179.641227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1179.648233] Lustre: Skipped 1 previous similar message [ 1184.740050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1184.745968] Lustre: Skipped 1 previous similar message [ 1188.835418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1188.843076] Lustre: Skipped 1 previous similar message [ 1191.391396] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1191.625908] Lustre: server umount lustre-MDT0000 complete [ 1194.796171] LustreError: 66150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773669269 with bad export cookie 7752069651152469296 [ 1194.802384] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1194.805813] LustreError: 66150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1195.054667] Lustre: server umount lustre-MDT0001 complete [ 1202.108601] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 09:54:34 (1773669274) [ 1211.593737] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1244.647359] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 1254.267633] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1284.158237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1284.347472] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1284.370677] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1284.453378] Lustre: lustre-MDT0000: new disk, initializing [ 1284.518921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1284.528629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1287.356412] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1294.628109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1294.737114] Lustre: 73720:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1294.787204] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1294.791895] Lustre: Skipped 1 previous similar message [ 1294.894554] Lustre: lustre-MDT0001: new disk, initializing [ 1294.957394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1294.985775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1294.990709] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1297.910505] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1303.604411] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1306.573569] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1311.974575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1312.232995] Lustre: lustre-OST0000: new disk, initializing [ 1312.237748] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1312.320720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1317.607278] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1320.496749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1320.512918] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1320.613277] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1323.926287] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1328.421308] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1328.671727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1331.979843] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1332.162453] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1336.290033] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1336.290128] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1336.298320] LustreError: Skipped 2 previous similar messages [ 1336.300390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1336.310827] Lustre: Skipped 2 previous similar messages [ 1339.901574] Lustre: server umount lustre-OST0000 complete [ 1346.529690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1346.532224] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1346.539320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1346.551041] Lustre: Skipped 1 previous similar message [ 1346.561115] Lustre: Skipped 2 previous similar messages [ 1348.751124] Lustre: server umount lustre-MDT0000 complete [ 1350.912753] LustreError: 73710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773669425 with bad export cookie 7752069651152470178 [ 1350.915497] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1350.915911] LustreError: 73710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1351.100338] Lustre: server umount lustre-MDT0001 complete [ 1356.621801] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 09:57:09 (1773669429) [ 1361.574344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1361.896659] LustreError: 77726:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1361.910184] LustreError: 77726:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 1361.949176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1365.222725] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1367.009016] LustreError: 77727:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1372.005819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1372.131856] LustreError: 77726:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1375.652403] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1379.653652] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1382.440619] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1387.823791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1392.724763] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1396.658715] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1399.230366] Lustre: 80060:0:(genops.c:1793:obd_export_evict_by_uuid()) MGS: evicting a07b7a93-54a3-469b-aaf9-183ce91f5323 at adminstrative request [ 1402.342845] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1402.354254] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 1412.582722] Lustre: MGS: Client a07b7a93-54a3-469b-aaf9-183ce91f5323 (at 0@lo) reconnecting [ 1412.587022] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x6b94e24687078414 to 0x6b94e246870786ad [ 1412.604558] Lustre: MGC192.168.203.128@tcp: Connection restored to 0@lo (at 0@lo) [ 1434.176883] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1434.308047] Lustre: lustre-OST0001: new disk, initializing [ 1434.312379] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1434.381195] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1434.385138] Lustre: Skipped 2 previous similar messages [ 1437.923391] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1440.361397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1440.370072] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1440.420724] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1442.868591] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1445.857753] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1445.863982] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.878057] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1445.885263] Lustre: Skipped 1 previous similar message [ 1446.708647] Lustre: server umount lustre-OST0001 complete [ 1450.992326] LustreError: 80074:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1451.004068] LustreError: 80074:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1455.394859] Lustre: server umount lustre-OST0000 complete [ 1459.681419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1459.688801] LustreError: Skipped 1 previous similar message [ 1459.690439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1459.697254] Lustre: Skipped 3 previous similar messages [ 1463.816372] Lustre: server umount lustre-MDT0000 complete [ 1466.150367] LustreError: 77707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773669540 with bad export cookie 7752069651152471725 [ 1466.155661] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1466.161199] LustreError: 77707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1466.405305] Lustre: server umount lustre-MDT0001 complete [ 1471.520691] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 09:59:04 (1773669544) [ 1472.429817] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 1473.300981] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 09:59:06 (1773669546) [ 1474.178529] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 1475.094404] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 09:59:08 (1773669548) [ 1475.887429] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 1476.893280] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 09:59:10 (1773669550) [ 1477.841153] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 1478.850911] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 09:59:11 (1773669551) [ 1479.654557] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 1482.773641] loop: module loaded [ 1482.780821] loop0: detected capacity change from 0 to 4294967296 [ 1482.813504] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 1483.848797] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 1485.657687] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 1487.740823] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 1500.493717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1500.519521] LustreError: 84441:0:(obd_config.c:880:class_setup()) setup lustre-OST0000-osd failed (-28) [ 1500.522969] LustreError: 84441:0:(obd_mount.c:217:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 1500.526749] LustreError: 84441:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 1500.531092] LustreError: 84441:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 1509.754807] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 1514.007569] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1530.659422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1530.802516] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1530.822342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1530.870909] Lustre: lustre-MDT0000: new disk, initializing [ 1530.917132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1530.928858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1532.800250] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1538.750813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1538.809644] Lustre: 87267:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1538.888174] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1538.893445] Lustre: Skipped 5 previous similar messages [ 1538.945789] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1540.831297] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1545.103890] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1547.206879] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1551.015049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1551.331044] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1554.296874] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1558.590765] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1560.492365] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1560.597311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1562.677316] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1562.775542] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1566.690214] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1566.694132] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1566.699641] Lustre: Skipped 1 previous similar message [ 1569.912195] Lustre: server umount lustre-OST0000 complete [ 1577.832788] Lustre: server umount lustre-MDT0000 complete [ 1579.290153] LustreError: 87259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773669653 with bad export cookie 7752069651152472348 [ 1579.293248] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1579.294413] LustreError: 87259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1579.423986] Lustre: server umount lustre-MDT0001 complete [ 1582.542775] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 10:00:56 (1773669656) [ 1585.030251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1585.203092] LustreError: 91272:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1586.626279] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1589.918884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1591.421306] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1593.422845] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1594.805334] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1597.306504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1599.401761] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1601.363266] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1602.788590] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1610.003240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 1611.310338] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1611.388868] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1613.537251] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1613.602418] Lustre: lustre-OST0001: new disk, initializing [ 1613.603779] Lustre: Skipped 2 previous similar messages [ 1613.605822] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1613.607594] Lustre: Skipped 3 previous similar messages [ 1615.446785] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1618.576123] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1618.922284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1618.926712] Lustre: Skipped 2 previous similar messages [ 1618.929327] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1618.931951] Lustre: Skipped 1 previous similar message [ 1618.946294] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1620.053674] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1620.132856] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1621.506657] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1621.587084] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1631.555263] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1636.364507] Lustre: Failing over lustre-MDT0000 [ 1636.737800] Lustre: server umount lustre-MDT0000 complete [ 1638.170464] LustreError: 91267:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1638.176778] LustreError: 91267:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1639.393941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1639.396871] Lustre: Skipped 6 previous similar messages [ 1651.399088] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 1651.401332] LDISKFS-fs (dm-0): recovery complete [ 1651.405853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1651.448681] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1651.576249] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1652.747949] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1653.524573] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 1653.525935] Lustre: 91297:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1656.803190] Lustre: 91273:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1656.805871] Lustre: 91273:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 1 previous similar message [ 1656.811934] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1656.825923] LustreError: 97015:0:(ldlm_lib.c:2680:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1657.143132] LustreError: 97015:0:(ldlm_lib.c:2680:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1657.457464] LustreError: 97015:0:(ldlm_lib.c:2680:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1657.461533] LustreError: 97015:0:(ldlm_lib.c:2680:replay_request_or_update()) Skipped 1 previous similar message [ 1657.775140] LustreError: 97015:0:(ldlm_lib.c:2680:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1657.777463] LustreError: 97015:0:(ldlm_lib.c:2680:replay_request_or_update()) Skipped 1 previous similar message [ 1658.097023] Lustre: 97015:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1658.100227] Lustre: 97015:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 8 previous similar messages [ 1658.728707] LustreError: 97015:0:(ldlm_lib.c:2680:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1658.733289] LustreError: 97015:0:(ldlm_lib.c:2680:replay_request_or_update()) Skipped 3 previous similar messages [ 1659.047130] LustreError: 97015:0:(ldlm_lib.c:2680:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1659.048968] LustreError: 97015:0:(ldlm_lib.c:2680:replay_request_or_update()) Skipped 3 previous similar messages [ 1660.303686] Lustre: 97015:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1660.307066] Lustre: 97015:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 6 previous similar messages [ 1660.936137] LustreError: 97015:0:(ldlm_lib.c:2680:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1660.939106] LustreError: 97015:0:(ldlm_lib.c:2680:replay_request_or_update()) Skipped 6 previous similar messages [ 1661.255144] LustreError: 97015:0:(ldlm_lib.c:2680:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1661.258823] LustreError: 97015:0:(ldlm_lib.c:2680:replay_request_or_update()) Skipped 6 previous similar messages [ 1664.385359] Lustre: 97015:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1664.389084] Lustre: 97015:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 12 previous similar messages [ 1665.016904] LustreError: 97015:0:(ldlm_lib.c:2680:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1665.019141] LustreError: 97015:0:(ldlm_lib.c:2680:replay_request_or_update()) Skipped 12 previous similar messages [ 1665.335126] LustreError: 97015:0:(ldlm_lib.c:2680:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1665.338135] LustreError: 97015:0:(ldlm_lib.c:2680:replay_request_or_update()) Skipped 12 previous similar messages [ 1672.520859] Lustre: 97015:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1672.523580] Lustre: 97015:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 25 previous similar messages [ 1673.152897] LustreError: 97015:0:(ldlm_lib.c:2680:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1673.155428] LustreError: 97015:0:(ldlm_lib.c:2680:replay_request_or_update()) Skipped 25 previous similar messages [ 1673.471079] LustreError: 97015:0:(ldlm_lib.c:2680:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1673.472995] LustreError: 97015:0:(ldlm_lib.c:2680:replay_request_or_update()) Skipped 25 previous similar messages [ 1688.776790] Lustre: 97015:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1688.780052] Lustre: 97015:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 51 previous similar messages [ 1689.408742] LustreError: 97015:0:(ldlm_lib.c:2680:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1689.410821] LustreError: 97015:0:(ldlm_lib.c:2680:replay_request_or_update()) Skipped 51 previous similar messages [ 1689.727100] LustreError: 97015:0:(ldlm_lib.c:2680:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1689.729413] LustreError: 97015:0:(ldlm_lib.c:2680:replay_request_or_update()) Skipped 51 previous similar messages [ 1713.135085] Lustre: 97015:0:(ldlm_lib.c:2376:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 1713.138832] Lustre: 97015:0:(ldlm_lib.c:2386:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1713.141017] Lustre: 97015:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client b7dd8468-0393-4919-8a86-813e3eccc176@192.168.203.28@tcp [ 1713.144272] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1713.146407] Lustre: 97015:0:(ldlm_lib.c:1896:abort_req_replay_queue()) @@@ aborted: req@000000007bccd54c x1859826886969856/t0(8589934777) o101->b7dd8468-0393-4919-8a86-813e3eccc176@192.168.203.28@tcp:393/0 lens 592/0 e 0 to 0 dl 1773669798 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 1713.155364] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1713.158509] Lustre: lustre-MDT0000: Denying connection for new client b7dd8468-0393-4919-8a86-813e3eccc176 (at 192.168.203.28@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 1713.164913] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 1713.183330] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1713.183523] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 1713.187547] Lustre: Skipped 2 previous similar messages [ 1713.205681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:79 to 0x280000401:577) [ 1713.205739] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:47 to 0x2c0000401:545) [ 1718.244296] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1718.244470] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1718.246530] LustreError: Skipped 1 previous similar message [ 1718.250570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1718.253269] Lustre: Skipped 7 previous similar messages [ 1718.970174] Lustre: server umount lustre-OST0000 complete [ 1723.362363] LustreError: 92879:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1723.366245] LustreError: 92879:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 17 previous similar messages [ 1726.287583] Lustre: server umount lustre-OST0001 complete [ 1733.668619] Lustre: server umount lustre-MDT0000 complete [ 1734.838617] LustreError: 92002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773669809 with bad export cookie 7752069651152523546 [ 1734.841823] LustreError: 92002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1734.845029] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1734.962375] Lustre: server umount lustre-MDT0001 complete [ 1737.592985] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 10:03:31 (1773669811) [ 1740.073034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1740.075159] Lustre: *** cfs_fail_loc=197, val=0*** [ 1755.487761] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1760.607576] LustreError: 98973:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.128@tcp: failed processing log, type 4: rc = -110 [ 1765.735732] Lustre: *** cfs_fail_loc=197, val=0*** [ 1765.736968] Lustre: Skipped 65 previous similar messages [ 1786.207513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1786.209934] Lustre: Skipped 7 previous similar messages [ 1787.985082] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1789.695595] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1790.423492] Lustre: server umount lustre-OST0000 complete [ 1793.049292] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 10:04:26 (1773669866) [ 1806.356290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1806.509806] LustreError: 100474:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1807.701325] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1810.600064] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1812.506816] LustreError: 100508:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1812.509700] LustreError: 100508:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1812.511771] LustreError: 100508:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 1812.632649] Lustre: server umount lustre-MDT0000 complete [ 1822.650773] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 1825.785331] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1836.806646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1836.897700] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1836.910246] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1836.948450] Lustre: lustre-MDT0000: new disk, initializing [ 1836.979971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1838.213418] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1842.141164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1842.179402] Lustre: 104525:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1842.278701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1843.428798] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1846.349884] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1847.544441] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1849.746155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1851.664797] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1851.749330] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1854.739996] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1856.091098] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1856.163354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1857.491463] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1857.573276] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1860.303509] Lustre: server umount lustre-OST0000 complete [ 1862.114332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1862.115090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1862.117427] Lustre: Skipped 6 previous similar messages [ 1862.119166] Lustre: Skipped 7 previous similar messages [ 1871.356927] Lustre: server umount lustre-MDT0000 complete [ 1872.352464] LustreError: 104537:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1872.357607] LustreError: 104537:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1872.559129] LustreError: 104519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773669946 with bad export cookie 7752069651152534207 [ 1872.561943] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1872.562917] LustreError: 104519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1883.153639] Lustre: server umount lustre-MDT0001 complete [ 1886.085756] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 10:05:59 (1773669959) [ 1889.062065] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1890.133221] Key type lgssc unregistered [ 1890.257366] LNet: 108586:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1890.259783] LNetError: 108586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1890.265347] LNet: Removed LNI 192.168.203.128@tcp [ 1890.582341] Key type .llcrypt unregistered [ 1890.583341] Key type ._llcrypt unregistered [ 1896.494687] Key type ._llcrypt registered [ 1896.495692] Key type .llcrypt registered [ 1896.532513] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 1902.788538] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1903.190903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1903.230721] alg: No test for adler32 (adler32-zlib) [ 1904.086607] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 1904.180906] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1905.767174] Key type lgssc registered [ 1906.171745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1919.409832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1919.416578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1920.512043] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1920.523318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1920.563300] Lustre: lustre-MDT0000: new disk, initializing [ 1920.587959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1920.594201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1921.927686] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1926.023174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1926.153409] Lustre: lustre-OST0000: new disk, initializing [ 1926.155494] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1926.157482] Lustre: Skipped 1 previous similar message [ 1926.182841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1928.123983] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1929.728757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1929.732848] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1929.741310] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1932.221795] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1932.282174] Lustre: lustre-OST0001: new disk, initializing [ 1932.284115] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1932.311373] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1934.215080] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1935.619079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 1935.622483] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 1935.631681] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 1940.960747] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1940.961270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1940.964372] Lustre: Skipped 1 previous similar message [ 1940.967605] Lustre: Skipped 1 previous similar message [ 1944.370065] Lustre: server umount lustre-MDT0000 complete [ 1946.542623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1948.113766] LustreError: 112679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773670022 with bad export cookie 1376222579916877834 [ 1948.116305] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1948.117303] LustreError: 112679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1948.148435] Lustre: server umount lustre-OST0000 complete [ 1949.433376] Lustre: server umount lustre-OST0001 complete [ 1952.242068] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 10:07:05 (1773670025) [ 1955.336959] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1962.397195] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 10:07:15 (1773670035) [ 1973.930051] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 1976.744819] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1987.230849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1987.313090] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1987.322507] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1987.354438] Lustre: lustre-MDT0000: new disk, initializing [ 1987.376077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1987.381226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1988.574867] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1992.525731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1992.563505] Lustre: 120196:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1992.582863] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1992.584562] Lustre: Skipped 1 previous similar message [ 1992.621266] Lustre: lustre-MDT0001: new disk, initializing [ 1992.654868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1992.666045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1992.668494] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1993.884045] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1996.774544] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1997.830676] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1999.928801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2001.662788] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2001.716858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2001.719520] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2001.759742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2004.466811] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2005.648103] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2005.707401] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2006.883724] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2006.949378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2012.129025] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2012.130516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2012.131504] LustreError: Skipped 1 previous similar message [ 2012.135953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2012.138622] Lustre: Skipped 1 previous similar message [ 2013.649188] Lustre: server umount lustre-OST0000 complete [ 2017.248867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2017.249651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2017.253085] Lustre: Skipped 2 previous similar messages [ 2017.256681] Lustre: Skipped 1 previous similar message [ 2020.958781] Lustre: server umount lustre-MDT0000 complete [ 2022.157110] LustreError: 120974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773670096 with bad export cookie 1376222579916879045 [ 2022.160150] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2022.160588] LustreError: 120974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2022.265867] Lustre: server umount lustre-MDT0001 complete [ 2024.866287] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 10:08:18 (1773670098) [ 2027.038347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2027.213940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2027.216643] Lustre: Skipped 1 previous similar message [ 2028.493185] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2031.288956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2032.709888] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2034.457823] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2035.598384] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2037.823226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2039.709751] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2041.569172] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2059.026098] Lustre: *** cfs_fail_loc=159, val=0*** [ 2075.422251] Lustre: lustre-MDT0000: Client 9810e9f6-d624-426c-a30e-d2fb31a31fa9 (at 192.168.203.28@tcp) reconnecting [ 2075.990891] Lustre: *** cfs_fail_loc=159, val=0*** [ 2075.993108] Lustre: Skipped 5 previous similar messages [ 2092.319420] Lustre: lustre-MDT0000: Client 9810e9f6-d624-426c-a30e-d2fb31a31fa9 (at 192.168.203.28@tcp) reconnecting [ 2094.563279] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2094.564695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2094.568373] Lustre: Skipped 1 previous similar message [ 2094.573323] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2099.542043] Lustre: server umount lustre-OST0000 complete [ 2103.263744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2103.266677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2103.271853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2103.273485] Lustre: Skipped 1 previous similar message [ 2115.039112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2115.040480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2115.045533] Lustre: Skipped 5 previous similar messages [ 2115.105975] Lustre: server umount lustre-MDT0000 complete [ 2116.421380] LustreError: 124181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773670190 with bad export cookie 1376222579916879913 [ 2116.425273] LustreError: 124181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2116.425407] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2116.563180] Lustre: server umount lustre-MDT0001 complete [ 2120.079238] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 2121.208383] Key type lgssc unregistered [ 2121.352445] LNet: 127638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2121.355072] LNetError: 127638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2121.362411] LNet: Removed LNI 192.168.203.128@tcp [ 2121.711136] Key type .llcrypt unregistered [ 2121.712354] Key type ._llcrypt unregistered [ 2131.025413] Key type ._llcrypt registered [ 2131.026494] Key type .llcrypt registered [ 2131.071410] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2131.429878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2131.450661] alg: No test for adler32 (adler32-zlib) [ 2132.313168] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 2132.402231] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 2133.983158] Key type lgssc registered [ 2134.330908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2136.908710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2136.914128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2138.095912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2139.420593] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2142.216026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2142.350616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2143.668715] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2145.402844] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2146.536932] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2148.758502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2148.834584] LustreError: 130606:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2148.839916] LustreError: 130606:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2148.871802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2150.131659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 2150.808307] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2152.566670] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2158.072372] Lustre: *** cfs_fail_loc=159, val=0*** [ 2174.238906] Lustre: lustre-MDT0000: Client 679a4d67-4dcc-49e2-bbd0-f7b0f80823f7 (at 192.168.203.28@tcp) reconnecting [ 2174.955390] Lustre: *** cfs_fail_loc=159, val=0*** [ 2174.957802] Lustre: Skipped 2 previous similar messages [ 2190.112408] Lustre: lustre-MDT0001: Client 679a4d67-4dcc-49e2-bbd0-f7b0f80823f7 (at 192.168.203.28@tcp) reconnecting [ 2190.646159] Lustre: *** cfs_fail_loc=159, val=0*** [ 2190.648638] Lustre: Skipped 1 previous similar message [ 2207.012855] Lustre: lustre-MDT0001: Client 679a4d67-4dcc-49e2-bbd0-f7b0f80823f7 (at 192.168.203.28@tcp) reconnecting [ 2208.178247] Lustre: 131714:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2208.964715] Lustre: *** cfs_fail_loc=159, val=0*** [ 2208.966177] Lustre: Skipped 4 previous similar messages [ 2224.416073] Lustre: lustre-MDT0000: Client efc2abc0-9c52-4590-ac14-ed4a72b7a991 (at 192.168.203.28@tcp) reconnecting [ 2224.946111] Lustre: *** cfs_fail_loc=159, val=0*** [ 2224.947804] Lustre: Skipped 14 previous similar messages [ 2240.290219] Lustre: lustre-MDT0000: Client efc2abc0-9c52-4590-ac14-ed4a72b7a991 (at 192.168.203.28@tcp) reconnecting [ 2240.775762] Lustre: 132004:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2242.527893] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2242.530894] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2242.535643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2244.579819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2244.583073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2244.588160] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2248.081467] Lustre: server umount lustre-OST0000 complete [ 2249.696379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2249.697081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2249.700175] Lustre: Skipped 1 previous similar message [ 2249.701868] Lustre: Skipped 1 previous similar message [ 2254.816182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2254.818839] Lustre: Skipped 2 previous similar messages [ 2255.509756] Lustre: server umount lustre-MDT0000 complete [ 2256.711123] LustreError: 128995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773670331 with bad export cookie 7221776900726738207 [ 2256.713893] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2256.714862] LustreError: 128995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2256.829114] Lustre: server umount lustre-MDT0001 complete [ 2260.368925] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 2261.360429] Key type lgssc unregistered [ 2261.490410] LNet: 133036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2261.494564] LNetError: 133036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2261.507332] LNet: Removed LNI 192.168.203.128@tcp [ 2261.806609] Key type .llcrypt unregistered [ 2261.807669] Key type ._llcrypt unregistered [ 2270.864927] Key type ._llcrypt registered [ 2270.865930] Key type .llcrypt registered [ 2270.900620] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2271.177455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2271.185196] alg: No test for adler32 (adler32-zlib) [ 2272.042429] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 2272.135920] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 2273.727136] Key type lgssc registered [ 2274.152372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2276.887381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2276.893032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2278.053107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2279.298305] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2282.032385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2282.146983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2283.343660] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2285.019297] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2286.181621] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2288.373780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2288.507149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2289.513353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 2290.386146] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2292.146307] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2292.583289] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 2295.330781] Lustre: 136658:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2295.992515] Lustre: 136707:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2296.486335] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 2297.648550] Lustre: 136853:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 2297.953341] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 2297.955940] Lustre: Skipped 1 previous similar message [ 2298.253457] Lustre: 136951:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2299.871828] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2299.873797] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2299.877151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2302.951126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2302.951402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2302.956763] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2305.292360] Lustre: server umount lustre-OST0000 complete [ 2308.064868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2308.064957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2308.066151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2308.066157] Lustre: Skipped 1 previous similar message [ 2308.068242] Lustre: Skipped 1 previous similar message [ 2312.671358] Lustre: server umount lustre-MDT0000 complete [ 2313.184399] LustreError: 134411:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2313.188440] LustreError: 134411:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 2313.943502] LustreError: 135128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773670388 with bad export cookie 7444116156639062041 [ 2313.948410] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2313.948683] LustreError: 135128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2314.077506] Lustre: server umount lustre-MDT0001 complete [ 2317.698582] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 2318.784254] Key type lgssc unregistered [ 2318.906420] LNet: 137993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2318.909395] LNetError: 137993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2318.921342] LNet: Removed LNI 192.168.203.128@tcp [ 2319.220290] Key type .llcrypt unregistered [ 2319.221609] Key type ._llcrypt unregistered [ 2327.909368] Key type ._llcrypt registered [ 2327.910352] Key type .llcrypt registered [ 2327.953938] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2328.336591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2328.394084] alg: No test for adler32 (adler32-zlib) [ 2329.256065] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 2329.353281] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 2330.943127] Key type lgssc registered [ 2331.361322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2333.978460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2333.984353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2335.143889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2336.494936] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2339.360053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2339.490961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2340.736496] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2342.526753] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2343.969927] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2346.762255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2346.943097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2348.884289] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2349.031420] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 2350.865635] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2360.295930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 2363.756177] Lustre: *** cfs_fail_loc=159, val=0*** [ 2363.757566] Lustre: Skipped 3 previous similar messages [ 2379.042122] Lustre: lustre-MDT0000: Client 8489038b-9dc3-4391-95d4-38d6fdf1a481 (at 192.168.203.28@tcp) reconnecting [ 2395.103916] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2395.106465] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2395.110494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2396.130321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2396.133183] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2396.134596] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2400.976766] Lustre: server umount lustre-OST0000 complete [ 2406.369553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2406.369709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2406.370734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2406.370741] Lustre: Skipped 1 previous similar message [ 2406.373453] Lustre: Skipped 1 previous similar message [ 2408.474667] Lustre: server umount lustre-MDT0000 complete [ 2409.673037] LustreError: 139346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773670484 with bad export cookie 16281837473974830017 [ 2409.676701] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2409.678157] LustreError: 139346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2409.788335] Lustre: server umount lustre-MDT0001 complete [ 2413.479353] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 2414.492938] Key type lgssc unregistered [ 2414.612385] LNet: 142696:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2414.614804] LNetError: 142696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2414.623311] LNet: Removed LNI 192.168.203.128@tcp [ 2414.924319] Key type .llcrypt unregistered [ 2414.925529] Key type ._llcrypt unregistered [ 2424.095207] Key type ._llcrypt registered [ 2424.096239] Key type .llcrypt registered [ 2424.130638] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2424.440117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2424.447069] alg: No test for adler32 (adler32-zlib) [ 2425.292981] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 2425.374101] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 2426.951163] Key type lgssc registered [ 2427.313254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2429.847205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2429.852886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2431.008896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2432.305226] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2435.118748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2435.267168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2436.591593] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2438.405993] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2439.579776] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2441.771316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2441.908704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2443.696165] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2444.966371] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 2445.486511] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2457.259243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 2473.731663] Lustre: lustre-OST0000: evicting d942c634-2cf8-4a62-baa0-2ba56d4a6adb (at 192.168.203.28@tcp) by administrative request [ 2473.734147] Lustre: Skipped 1 previous similar message [ 2498.530614] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2498.531053] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2498.531550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2498.535677] Lustre: Skipped 1 previous similar message [ 2503.648355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2503.651153] Lustre: Skipped 2 previous similar messages [ 2508.768562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2508.770463] Lustre: Skipped 1 previous similar message [ 2512.351104] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2512.402994] Lustre: server umount lustre-OST0000 complete [ 2513.888993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2513.889722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2513.893018] Lustre: Skipped 1 previous similar message [ 2513.897181] Lustre: Skipped 1 previous similar message [ 2519.009781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2519.012200] Lustre: Skipped 1 previous similar message [ 2519.768811] Lustre: server umount lustre-MDT0000 complete [ 2521.095916] LustreError: 144050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773670595 with bad export cookie 12132396492596349717 [ 2521.097157] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2521.101381] LustreError: 144050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2521.223135] Lustre: server umount lustre-MDT0001 complete [ 2525.081987] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 2526.123716] Key type lgssc unregistered [ 2526.246357] LNet: 147809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2526.248770] LNetError: 147809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2526.258320] LNet: Removed LNI 192.168.203.128@tcp [ 2526.541408] Key type .llcrypt unregistered [ 2526.542619] Key type ._llcrypt unregistered [ 2551.879083] Key type ._llcrypt registered [ 2551.880121] Key type .llcrypt registered [ 2551.919661] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2552.239485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2552.294843] alg: No test for adler32 (adler32-zlib) [ 2553.154461] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 2553.241456] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 2554.823140] Key type lgssc registered [ 2555.169449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2557.706200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2557.712397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2558.888192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2560.239097] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2563.083821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2563.222515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2564.566719] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2566.343989] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2567.540214] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2569.851288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2569.976232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2571.800413] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2573.034881] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 2573.524579] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2585.318504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2593.759953] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2593.763463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2593.769795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2595.809762] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2595.810193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2595.810309] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2595.816580] Lustre: Skipped 1 previous similar message [ 2597.778747] Lustre: server umount lustre-OST0000 complete [ 2599.391901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2599.394167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2599.397964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2605.150304] Lustre: server umount lustre-MDT0000 complete [ 2606.048052] LustreError: 150252:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2606.052597] LustreError: 150252:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 2606.362151] LustreError: 150232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773670680 with bad export cookie 12134801854553732298 [ 2606.365897] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2606.366805] LustreError: 150232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2606.475178] Lustre: server umount lustre-MDT0001 complete [ 2609.825131] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 2610.756447] Key type lgssc unregistered [ 2610.875385] LNet: 153459:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2610.877587] LNetError: 153459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2610.887300] LNet: Removed LNI 192.168.203.128@tcp [ 2611.163128] Key type .llcrypt unregistered [ 2611.163995] Key type ._llcrypt unregistered [ 2620.293740] Key type ._llcrypt registered [ 2620.295082] Key type .llcrypt registered [ 2620.331426] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2620.725519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2620.732040] alg: No test for adler32 (adler32-zlib) [ 2621.588036] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 2621.675246] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 2623.255137] Key type lgssc registered [ 2623.612031] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2628.670963] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 10:18:22 (1773670702) [ 2634.295376] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 2637.367973] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2649.343100] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2649.349055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2650.465329] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2650.479942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2650.519890] Lustre: lustre-MDT0000: new disk, initializing [ 2650.550649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2650.557667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2651.882946] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2655.957733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2655.999600] Lustre: 157640:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2656.032452] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2656.033916] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2656.035887] Lustre: Skipped 1 previous similar message [ 2656.076468] Lustre: lustre-MDT0001: new disk, initializing [ 2656.111349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2656.125483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2656.128230] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2657.392149] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2660.431908] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2661.725867] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2664.034231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2664.142323] Lustre: lustre-OST0000: new disk, initializing [ 2664.143996] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2664.175913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2664.950789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2664.953691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2665.000127] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2666.090826] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2669.122156] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2675.171166] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2675.171786] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2675.179666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2675.183660] LustreError: Skipped 1 previous similar message [ 2675.187891] Lustre: Skipped 1 previous similar message [ 2678.355095] Lustre: server umount lustre-OST0000 complete [ 2680.288731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2680.289793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2680.292572] Lustre: Skipped 2 previous similar messages [ 2680.297308] Lustre: Skipped 1 previous similar message [ 2685.408927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2685.412895] Lustre: Skipped 1 previous similar message [ 2690.528776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2690.531427] Lustre: Skipped 1 previous similar message [ 2694.111162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2694.204347] Lustre: server umount lustre-MDT0000 complete [ 2695.791320] LustreError: 157632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773670770 with bad export cookie 12228281934877072821 [ 2695.794239] LustreError: 157632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2695.794352] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2695.915436] Lustre: server umount lustre-MDT0001 complete [ 2699.907347] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 2701.098322] Key type lgssc unregistered [ 2701.239540] LNet: 161156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2701.241836] LNetError: 161156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2701.253465] LNet: Removed LNI 192.168.203.128@tcp [ 2701.607238] Key type .llcrypt unregistered [ 2701.608303] Key type ._llcrypt unregistered [ 2711.093451] Key type ._llcrypt registered [ 2711.094455] Key type .llcrypt registered [ 2711.130179] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2711.534049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2711.554367] alg: No test for adler32 (adler32-zlib) [ 2712.405532] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 2712.488668] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 2714.071139] Key type lgssc registered [ 2714.409728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2717.231322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2717.237287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2718.390193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2719.778413] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2722.732540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2722.860313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2724.127291] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2725.882637] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2727.092657] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2729.312711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2729.423611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2731.241599] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2733.041693] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2743.784807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2750.969150] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2750.976391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2750.980028] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2750.982826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2750.986201] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2751.291940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2751.613864] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2751.615710] Lustre: Skipped 1 previous similar message [ 2751.620148] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2751.623777] Lustre: Skipped 1 previous similar message [ 2751.625035] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2751.626721] Lustre: Skipped 1 previous similar message [ 2751.627836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2751.632326] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2751.634741] Lustre: Skipped 1 previous similar message [ 2751.930488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2752.223268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2752.534534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2752.855475] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2752.857617] Lustre: Skipped 3 previous similar messages [ 2752.862376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2752.865739] Lustre: Skipped 3 previous similar messages [ 2752.867167] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2752.868935] Lustre: Skipped 3 previous similar messages [ 2752.870127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2752.873368] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2752.875069] Lustre: Skipped 3 previous similar messages [ 2753.161359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2753.464207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2753.799086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2754.147971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2754.461974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2754.774787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2755.083625] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2755.085472] Lustre: Skipped 6 previous similar messages [ 2755.090762] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2755.097011] Lustre: Skipped 6 previous similar messages [ 2755.098433] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2755.099989] Lustre: Skipped 6 previous similar messages [ 2755.101293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2755.103948] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2755.105375] Lustre: Skipped 6 previous similar messages [ 2755.409539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2755.698809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2755.983247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2756.314321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2756.638633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2756.940540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2757.248393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2757.558501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2757.872673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2758.162130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2758.480159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2758.785235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2759.095950] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2759.097848] Lustre: Skipped 12 previous similar messages [ 2759.103360] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2759.106875] Lustre: Skipped 12 previous similar messages [ 2759.109079] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2759.111015] Lustre: Skipped 12 previous similar messages [ 2759.112846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2759.116572] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2759.118128] Lustre: Skipped 12 previous similar messages [ 2759.411260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2759.721307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2760.004940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2760.316486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2760.660617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2760.980527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2761.316551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2761.655105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2762.022154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2762.360831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2762.677854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2763.001964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2763.337310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2763.675861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2764.014309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2764.339491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2764.660830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2764.997217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2765.335512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2765.728593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2766.099716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2766.398700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2766.723839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2767.056261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2767.413163] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2767.417044] Lustre: Skipped 24 previous similar messages [ 2767.427386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2767.434564] Lustre: Skipped 24 previous similar messages [ 2767.437810] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2767.441079] Lustre: Skipped 24 previous similar messages [ 2767.442941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2767.451818] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2767.456386] Lustre: Skipped 24 previous similar messages [ 2767.790040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2768.113501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2768.443820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2768.847409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2769.257432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2769.568517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2769.896045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2770.238192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2770.582683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2770.915791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2771.236607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2771.552750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2771.902869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2772.220245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2772.541767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2772.850623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2773.165895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2773.453589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2773.752650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2774.064562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2774.348530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2774.631448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2774.943624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2775.283170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2775.582135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2775.886437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2776.230977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2776.559471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2776.877112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2777.176678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2777.496509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2777.781088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2778.104199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2778.421722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2778.763144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2779.156938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2779.495920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2779.874618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2780.261088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2780.648439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2781.073870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2781.506187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2781.945426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2782.366311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2782.812752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2783.253492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2783.659837] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2783.663063] Lustre: Skipped 46 previous similar messages [ 2783.670233] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2783.675891] Lustre: Skipped 46 previous similar messages [ 2783.678287] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2783.681363] Lustre: Skipped 46 previous similar messages [ 2783.683558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2783.693373] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2783.696233] Lustre: Skipped 46 previous similar messages [ 2784.108493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2784.514354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2784.925551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2785.329970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2785.774927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2786.217567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2786.710079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2787.125963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2787.515441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2787.986602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2788.387233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2788.809249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2789.151375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2789.487488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2789.831207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2790.188081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2790.502461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2790.833409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2791.191099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2791.585589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2791.943084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2792.355978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2792.691507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2793.018270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2793.360625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2793.680820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2793.969653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2794.236886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2794.513094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2794.846350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2795.233855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2795.624195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2795.932506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2796.296355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2796.676154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2797.032271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2797.320942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2797.673132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2798.030736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2798.340546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2798.664339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2798.974303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2799.322133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2799.635646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2799.959620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2800.275555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2800.611948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2800.929212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2801.317290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2801.672442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2802.029622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2802.385465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2802.727282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2803.041609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2803.338059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2803.646464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2803.968598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2804.257803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2804.612611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2805.002638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2805.417305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2805.787932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2806.130709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2806.491432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2806.790976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2807.084574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2807.372982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2807.776830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2808.212345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2808.514470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2808.790432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2809.077759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2809.360920] LustreError: 162550:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2809.363678] LustreError: 162550:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2809.364847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2809.632806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2809.965809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2810.381502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2810.778953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2811.131492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2811.466234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2811.775426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2812.169113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2812.536062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2812.918929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2813.318961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2813.650110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2813.978521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2814.400448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2814.859638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2815.212072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2815.561327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2815.952892] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2815.958217] Lustre: Skipped 90 previous similar messages [ 2815.968785] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2815.976853] Lustre: Skipped 90 previous similar messages [ 2815.980139] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2815.984586] Lustre: Skipped 90 previous similar messages [ 2815.987752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2815.994930] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2815.999057] Lustre: Skipped 90 previous similar messages [ 2816.447457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2816.907366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2817.244557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2817.657284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2818.110042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2818.567819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2818.903783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2819.314382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2819.738432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2820.200605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2820.573835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2820.998543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2821.367804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2821.796353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2822.199445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2822.588409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2823.018537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2823.455709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2823.913773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2824.338462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2824.736481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2825.165573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2825.612915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2826.046108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2826.463995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2826.890511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2827.344538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2827.787923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2828.180388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2828.587573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2828.988846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2829.352333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2829.729773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2830.058578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2830.417807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2830.800532] LustreError: 162550:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2830.804167] LustreError: 162550:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2830.805488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2831.102596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2831.427358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2831.749300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2832.094361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2832.432529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2832.749971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2833.149457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2833.503644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2833.837755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2834.222797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2834.655341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2835.034880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2835.400171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2835.758571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2836.129853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2836.409788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2836.707996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2837.009118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2837.295531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2837.577685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2837.869582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2838.174361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2838.531138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2838.853294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2839.182185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2839.555228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2839.872305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2840.188909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2840.521957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2840.930170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2841.270349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2841.701734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2842.095451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2842.537952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2842.938551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2843.385824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2843.823464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2844.274270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2844.721529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2845.087826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2845.492879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2845.866927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2846.177435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2846.463263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2846.836695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2847.253957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2847.673546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2848.118131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2848.504614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2848.898799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2849.294670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2849.575980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2849.872368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2850.216660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2850.572750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2851.008456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2851.492325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2851.914164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2852.333124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2852.670770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2852.970327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2853.260797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2853.551539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2853.848885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2854.161035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2854.455824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2854.750861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2855.051227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2855.371346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2855.654762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2855.962867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2856.265827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2856.593874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2856.891708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2857.202877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2857.504512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2857.804433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2858.092669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2858.381197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2858.659312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2858.938205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2859.230737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2859.582050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2859.918516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2860.378771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2860.717170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2861.016778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2861.344639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2861.668655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2861.993847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2862.334305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2862.759349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2863.097312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2863.473385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2863.857298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2864.177965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2864.626206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2864.980188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2865.273746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2865.549285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2865.850900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2866.296108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2866.712175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2867.166307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2867.507113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2867.903731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2868.355768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2868.822641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2869.166861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2869.562366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2869.892857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2870.203573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2870.518802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2870.983716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2871.424868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2871.834637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2872.233664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2872.571680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2872.974552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2873.354070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2873.784891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2874.191094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2874.671349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2875.037485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2875.355560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2875.776788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2876.113263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2876.389375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2876.729568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2877.141419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2877.532996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2877.950050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2878.285944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2878.616937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2878.960310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2879.267046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2879.566056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2879.855441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2880.181749] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2880.187163] Lustre: Skipped 174 previous similar messages [ 2880.197207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2880.205206] Lustre: Skipped 174 previous similar messages [ 2880.208601] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2880.212969] Lustre: Skipped 174 previous similar messages [ 2880.216175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2880.226796] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2880.233120] Lustre: Skipped 174 previous similar messages [ 2880.695373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2881.018592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2881.293579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2881.636976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2882.073677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2882.420882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2882.826349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2883.235511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2883.617386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2884.015789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2884.418744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2884.800824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2885.206545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2885.527326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2885.861309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2886.202336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2886.554879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2886.855454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2887.130683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2887.409251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2887.723574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2888.021632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2888.439679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2888.894679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2889.356447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2889.685535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2890.012704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2890.331065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2890.656078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2891.004564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2891.336769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2891.666835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2892.050067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2892.391598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2892.773627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2893.073028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2893.393176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2893.709096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2894.044619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2894.391265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2894.813079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2895.234458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2895.600825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2896.021513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2896.391604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2896.807203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2897.161747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2897.482514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2897.809089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2898.151349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2898.456987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2898.801268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2899.146536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2899.495631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2899.828876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2900.164047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2900.548763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2900.918220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2901.277851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2901.738727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2902.216101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2902.692862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2903.149628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2903.522437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2903.911291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2904.319237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2904.674326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2905.034543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2905.348369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2905.698509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2906.021361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2906.349757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2906.672500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2907.044321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2907.383159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2907.759165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2908.101799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2908.481951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2908.802675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2909.233670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2909.564927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2909.890310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2910.257673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2910.602980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2910.966337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2911.288908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2911.636320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2911.949055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2912.273591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2912.586881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2912.907588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2913.231708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2913.566675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2913.901642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2914.197185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2914.485439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2914.767770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2915.067369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2915.378880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2915.686285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2915.999772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2916.317359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2916.609374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2916.904988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2917.196905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2917.523271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2917.830605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2918.125866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2918.429504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2918.736630] LustreError: 162550:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2918.739351] LustreError: 162550:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2918.741671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2919.041053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2919.338345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2919.647834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2919.961024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2920.236091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2920.533580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2920.869718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2921.204479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2921.524468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2921.826524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2922.190133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2922.545265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2922.838093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2923.153021] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2923.456455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2923.783438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2924.080655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2924.371021] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2924.670106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2924.979392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2925.275398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2925.596956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2925.916053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2926.246907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2926.549534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2926.865814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2927.217394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2927.560738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2927.855656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2928.168175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2928.461565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2928.835555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2929.202851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2929.501865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2929.830207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2930.267385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2930.690994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2931.112257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2931.539274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2931.980148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2932.375450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2932.781483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2933.247534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2933.673574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2934.137343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2934.524203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2934.932653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2935.338973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2935.720771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2936.131535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2936.485149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2936.862130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2937.274838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2937.718611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2938.200211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2938.662394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2939.112663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2939.496762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2939.952129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2940.422736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2940.790064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2941.084526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2941.450386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2941.798766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2942.088437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2942.400847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2942.705937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2943.021268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2943.313396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2943.642097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2943.940236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2944.244303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2944.521044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2944.833388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2945.146360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2945.488538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2945.799482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2946.103285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2946.394109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2946.692040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2946.998154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2947.304129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2947.605340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2947.924853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2948.231230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2948.543340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2948.864595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2949.194301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2949.509203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2949.810463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2950.123640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2950.435752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2950.743896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2951.045746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2951.343705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2951.666857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2951.965442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2952.271502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2952.581707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2952.897672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2953.199224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2953.514179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2953.817529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2954.130807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2954.430720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2954.735217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2955.035322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2955.345045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2955.639292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2955.939387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2956.228855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2956.532912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2956.826438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2957.129311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2957.419478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2957.733481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2958.024578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2958.322337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2958.622396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2958.934749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2959.238117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2959.521562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2959.822926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2960.140300] LustreError: 162550:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2960.142777] LustreError: 162550:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2960.145451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2960.433846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2960.734638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2961.042271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2961.344206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2961.628338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2961.940591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2962.242846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2962.564528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2962.872337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2963.182435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2963.482639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2963.811220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2964.123994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2964.430312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2964.729911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2965.051727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2965.332284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2965.604263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2965.872471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2966.187669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2966.491677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2966.778475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2967.062029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2967.349730] LustreError: 162550:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2967.352781] LustreError: 162550:0:(osp_precreate.c:651:osp_precreate_send()) Skipped 1 previous similar message [ 2967.353268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2967.355108] LustreError: 162550:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2967.361491] LustreError: 162550:0:(osp_precreate.c:1355:osp_precreate_thread()) Skipped 1 previous similar message [ 2967.663092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2967.972126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2968.277179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2968.577580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2968.877895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2969.179958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2969.480381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2988.515801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2988.519145] Lustre: Skipped 1 previous similar message [ 2991.628284] Lustre: server umount lustre-OST0000 complete [ 2993.634334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2993.636921] Lustre: Skipped 1 previous similar message [ 2998.752890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2998.755074] Lustre: Skipped 1 previous similar message [ 2999.025186] Lustre: server umount lustre-MDT0000 complete [ 3000.212071] LustreError: 162512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773671074 with bad export cookie 10264122089295392952 [ 3000.215422] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3000.217747] LustreError: 162512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3000.334882] Lustre: server umount lustre-MDT0001 complete [ 3003.988034] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 3004.992672] Key type lgssc unregistered [ 3005.137503] LNet: 198273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3005.139514] LNetError: 198273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3005.150281] LNet: Removed LNI 192.168.203.128@tcp [ 3005.444049] Key type .llcrypt unregistered [ 3005.444989] Key type ._llcrypt unregistered [ 3014.565385] Key type ._llcrypt registered [ 3014.566395] Key type .llcrypt registered [ 3014.602649] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3014.913610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3014.959525] alg: No test for adler32 (adler32-zlib) [ 3015.808923] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 3015.888097] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 3017.471119] Key type lgssc registered [ 3017.855503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3020.419347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3020.425391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3021.584526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3023.023168] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3026.273788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3026.411787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3027.831015] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3029.688776] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3031.167347] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3033.589565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3033.701472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3035.557582] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3037.307866] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3047.021550] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11572 to 0x280000401:11617) [ 3050.818871] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3051.445107] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3051.451536] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3051.455910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3051.458815] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3051.458848] LustreError: 199667:0:(osp_precreate.c:1263:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 3052.513815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3052.514710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3052.516653] Lustre: Skipped 1 previous similar message [ 3052.519139] Lustre: Skipped 1 previous similar message [ 3054.225034] Lustre: server umount lustre-OST0000 complete [ 3071.535177] LustreError: 199667:0:(osp_precreate.c:1263:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 3082.243717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3082.356913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3084.325807] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3086.433646] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3092.584087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3092.589693] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3092.595200] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3092.598425] Lustre: Skipped 1 previous similar message [ 3094.346202] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 7 sec [ 3095.877104] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9f37a030f800.ost_server_uuid 50 [ 3096.397752] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9f37a030f800.ost_server_uuid in FULL state after 0 sec [ 3098.085693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3098.094673] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3098.094850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3098.103412] Lustre: Skipped 2 previous similar messages [ 3103.202342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3103.205741] Lustre: Skipped 1 previous similar message [ 3103.581438] Lustre: server umount lustre-OST0000 complete [ 3108.322453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3108.323856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3108.324510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3108.329119] Lustre: Skipped 2 previous similar messages [ 3111.447629] Lustre: server umount lustre-MDT0000 complete [ 3112.689543] LustreError: 201261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773671187 with bad export cookie 7107529973791582797 [ 3112.692457] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3112.694267] LustreError: 201261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3112.816097] Lustre: server umount lustre-MDT0001 complete [ 3116.313059] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 3117.283675] Key type lgssc unregistered [ 3117.412415] LNet: 204313:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3117.416572] LNetError: 204313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3117.427283] LNet: Removed LNI 192.168.203.128@tcp [ 3117.717481] Key type .llcrypt unregistered [ 3117.718539] Key type ._llcrypt unregistered [ 3130.841323] Key type ._llcrypt registered [ 3130.842280] Key type .llcrypt registered [ 3130.882826] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3131.335586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3131.344541] alg: No test for adler32 (adler32-zlib) [ 3132.192792] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 3132.272533] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 3133.847100] Key type lgssc registered [ 3134.208071] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3137.868552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3137.880477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3139.124802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3140.686161] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3143.720600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3143.856995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3145.339634] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3146.517428] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3148.975118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3149.123143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3150.992045] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3153.940258] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3154.021290] Lustre: lustre-OST0001: new disk, initializing [ 3154.024413] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3154.065869] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3155.654981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3155.659798] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3155.703625] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3156.191739] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3161.826981] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3163.882612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11619 to 0x280000401:11649) [ 3167.541235] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3180.000056] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3180.004522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3180.011471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3184.608799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3184.611166] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3184.614854] Lustre: Skipped 2 previous similar messages [ 3185.903715] Lustre: server umount lustre-MDT0000 complete [ 3187.117142] LustreError: 208256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773671261 with bad export cookie 8009762774692021165 [ 3187.120133] LustreError: 208256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3187.121394] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3187.229268] Lustre: server umount lustre-MDT0001 complete [ 3197.953016] Lustre: server umount lustre-OST0000 complete [ 3209.513593] Lustre: server umount lustre-OST0001 complete [ 3216.944189] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3221.555151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3221.746522] LustreError: 211334:0:(ldlm_lib.c:1178:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3221.753859] LustreError: 211334:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3221.775056] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3223.568676] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3227.105322] LustreError: 211335:0:(ldlm_lib.c:1178:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3227.377131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3228.917062] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3229.841789] Lustre: 212443:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3231.989746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3232.101944] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3232.104333] Lustre: Skipped 1 previous similar message [ 3233.886774] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3236.781414] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3238.796638] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3247.088582] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11651 to 0x280000401:11681) [ 3247.092840] Lustre: mylustre-OST0001: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3252.462243] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3253.681919] Lustre: 214305:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3263.457241] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3263.461082] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3263.467301] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3263.469890] Lustre: Skipped 3 previous similar messages [ 3265.273523] Lustre: server umount mylustre-MDT0000 complete [ 3266.531413] LustreError: 211310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773671340 with bad export cookie 8009762774692023930 [ 3266.536903] LustreError: 211310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3266.538133] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3267.553339] Lustre: mylustre-MDT0000-lwp-OST0000: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3267.554894] LustreError: 214051:0:(ldlm_lib.c:1178:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3267.555081] Lustre: mylustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3267.557018] Lustre: Skipped 2 previous similar messages [ 3267.563714] LustreError: 214051:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3268.686861] Lustre: server umount mylustre-MDT0001 complete [ 3270.040106] Lustre: server umount mylustre-OST0000 complete [ 3271.295419] Lustre: server umount mylustre-OST0001 complete [ 3278.193197] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3281.602395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3281.782737] LustreError: 216597:0:(ldlm_lib.c:1178:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3281.808618] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3281.810820] Lustre: Skipped 1 previous similar message [ 3283.099443] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3285.909649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3287.338883] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3288.312957] Lustre: 217705:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3290.496633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3291.621055] LustreError: 218058:0:(ldlm_lib.c:1178:target_handle_connect()) tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3291.625400] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11683 to 0x280000401:11713) [ 3292.511644] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3295.262402] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3296.362430] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 3297.167338] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3300.256590] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3301.438470] Lustre: 219547:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3311.584117] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3311.587519] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3311.592499] Lustre: Skipped 1 previous similar message [ 3311.594693] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 3311.597167] Lustre: Skipped 1 previous similar message [ 3314.474908] Lustre: server umount tfs-MDT0000 complete [ 3315.697329] LustreError: 219549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773671390 with bad export cookie 8009762774692025820 [ 3315.699782] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3315.702287] LustreError: 219549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3315.817944] Lustre: server umount tfs-MDT0001 complete [ 3327.430979] Lustre: server umount tfs-OST0000 complete [ 3338.900172] Lustre: server umount tfs-OST0001 complete [ 3345.378698] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3350.644993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3350.910756] LustreError: 221779:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3350.989593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3350.994623] Lustre: Skipped 3 previous similar messages [ 3355.774640] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3369.811278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3375.246811] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3380.217423] Lustre: 222887:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3390.413641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3394.865254] LustreError: 223243:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3394.895349] LustreError: 223243:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 3395.971172] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11683 to 0x280000401:11745) [ 3399.690688] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3408.432253] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3410.513973] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 3412.485399] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3417.181961] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3419.178520] Lustre: 224731:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3427.991265] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 10:31:41 (1773671501) [ 3442.143951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3442.147329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3442.152570] Lustre: Skipped 3 previous similar messages [ 3442.155097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3442.157312] Lustre: Skipped 4 previous similar messages [ 3446.933184] Lustre: server umount lustre-MDT0000 complete [ 3448.894261] LustreError: 221757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773671523 with bad export cookie 8009762774692027479 [ 3448.895977] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3448.900996] LustreError: 221757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3449.824295] LustreError: 221776:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3449.829769] LustreError: 221776:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 7 previous similar messages [ 3455.138423] Lustre: server umount lustre-MDT0001 complete [ 3463.375255] Lustre: server umount lustre-OST0000 complete [ 3471.531128] Lustre: server umount lustre-OST0001 complete [ 3474.788762] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 3479.954696] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3498.389507] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 3498.389507] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 3498.389507] [ 3498.396193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 3498.518276] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3498.535715] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3498.577628] Lustre: lustre-MDT0000: new disk, initializing [ 3498.605534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3498.607683] Lustre: Skipped 3 previous similar messages [ 3498.615237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3500.262272] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3505.518379] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 3505.518379] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 3505.518379] [ 3505.534460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 3505.609693] Lustre: 227883:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3505.641611] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3505.643659] Lustre: Skipped 1 previous similar message [ 3505.689225] Lustre: lustre-MDT0001: new disk, initializing [ 3505.761749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3505.768622] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3507.741715] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3512.940477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3513.051695] Lustre: lustre-OST0000: new disk, initializing [ 3513.054340] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3515.049421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3515.056364] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3515.123300] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3515.927600] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3522.037441] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3522.141962] Lustre: lustre-OST0001: new disk, initializing [ 3522.144184] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3523.534873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3523.538911] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3523.589360] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3525.040304] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3531.465672] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 10:33:24 (1773671604) [ 3538.708641] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 10:33:31 (1773671611) [ 3551.203294] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3551.203979] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3551.210243] LustreError: Skipped 1 previous similar message [ 3551.211674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3551.211680] Lustre: Skipped 8 previous similar messages [ 3551.237330] Lustre: Skipped 6 previous similar messages [ 3557.233352] Lustre: server umount lustre-OST0000 complete [ 3559.393684] LustreError: 229675:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3559.407415] LustreError: 229675:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3567.507629] Lustre: server umount lustre-MDT0000 complete [ 3569.948306] LustreError: 229431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773671644 with bad export cookie 8009762774692029040 [ 3569.954440] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3569.955347] LustreError: 229431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3576.288337] Lustre: server umount lustre-MDT0001 complete [ 3582.329671] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 3593.854896] Lustre: server umount lustre-OST0001 complete [ 3597.072146] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 3604.226127] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3616.267721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3616.350950] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3616.360467] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3616.391877] Lustre: lustre-MDT0000: new disk, initializing [ 3616.417304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3617.571239] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3621.571491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3621.612662] Lustre: 234744:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3621.717701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3622.909369] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3625.800690] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3626.892452] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3629.005614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3629.154838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3629.156913] Lustre: Skipped 5 previous similar messages [ 3630.138575] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3630.179946] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3631.039571] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3634.040141] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3640.289766] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3640.292416] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3640.293894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3640.296239] Lustre: Skipped 5 previous similar messages [ 3640.301416] Lustre: Skipped 11 previous similar messages [ 3642.319101] Lustre: server umount lustre-OST0000 complete [ 3649.756079] Lustre: server umount lustre-MDT0000 complete [ 3650.965123] LustreError: 236483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773671725 with bad export cookie 8009762774692030510 [ 3650.968089] LustreError: 236483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3650.968297] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3651.089668] Lustre: server umount lustre-MDT0001 complete [ 3654.491579] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 3655.449531] Key type lgssc unregistered [ 3655.565301] LNet: 238115:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3655.567347] LNetError: 238115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3655.576270] LNet: Removed LNI 192.168.203.128@tcp [ 3655.857124] Key type .llcrypt unregistered [ 3655.858818] Key type ._llcrypt unregistered [ 3665.450069] Key type ._llcrypt registered [ 3665.451595] Key type .llcrypt registered [ 3665.487239] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3665.861908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3665.894746] alg: No test for adler32 (adler32-zlib) [ 3666.741359] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 3666.821176] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 3668.399123] Key type lgssc registered [ 3668.741844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3671.241299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3671.247303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3672.401973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3673.641676] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3676.438047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3676.563729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3677.795115] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3679.574028] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3680.764861] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3683.007398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3683.122474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3684.919233] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3686.599277] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3686.934885] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 3691.489715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3691.490795] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3691.493778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3693.640060] Lustre: server umount lustre-OST0000 complete [ 3696.609035] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3696.613228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3696.614777] Lustre: Skipped 1 previous similar message [ 3700.883733] Lustre: server umount lustre-MDT0000 complete [ 3701.728154] LustreError: 239535:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3701.733149] LustreError: 239535:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 3702.068270] LustreError: 239517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773671776 with bad export cookie 4485642407617956476 [ 3702.071443] LustreError: 239517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3702.072694] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3702.186526] Lustre: server umount lustre-MDT0001 complete [ 3705.539134] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 3706.533040] Key type lgssc unregistered [ 3706.656380] LNet: 242763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3706.659247] LNetError: 242763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3706.668280] LNet: Removed LNI 192.168.203.128@tcp [ 3706.965401] Key type .llcrypt unregistered [ 3706.967364] Key type ._llcrypt unregistered [ 3713.577839] Key type ._llcrypt registered [ 3713.579538] Key type .llcrypt registered [ 3713.617383] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3714.021452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3714.028972] alg: No test for adler32 (adler32-zlib) [ 3714.875564] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 3714.953457] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 3716.527095] Key type lgssc registered [ 3716.826424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3719.632570] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3719.639471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3735.071218] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3740.191149] LustreError: 243917:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.128@tcp: failed processing log, type 4: rc = -110 [ 3766.815175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3768.638673] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3770.389180] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3771.112912] Lustre: server umount lustre-OST0000 complete [ 3776.275228] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 3777.292925] Key type lgssc unregistered [ 3777.408362] LNet: 245416:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3777.412657] LNetError: 245416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3777.420300] LNet: Removed LNI 192.168.203.128@tcp [ 3777.695659] Key type .llcrypt unregistered [ 3777.696664] Key type ._llcrypt unregistered [ 3791.442312] Key type ._llcrypt registered [ 3791.443267] Key type .llcrypt registered [ 3791.481094] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3791.797371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3791.827053] alg: No test for adler32 (adler32-zlib) [ 3792.669589] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 3792.743848] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 3794.319168] Key type lgssc registered [ 3794.671071] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3798.928254] loop0: detected capacity change from 0 to 204800000 [ 3798.989525] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3798.994403] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3799.114337] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3799.465721] loop0: detected capacity change from 0 to 204800000 [ 3799.527076] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3799.530342] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3799.658125] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3800.033899] loop0: detected capacity change from 0 to 204800000 [ 3800.103834] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3800.107803] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3800.153364] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3800.531697] loop0: detected capacity change from 0 to 204800000 [ 3800.586368] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3800.988514] loop0: detected capacity change from 0 to 204800000 [ 3800.999760] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3802.326590] loop0: detected capacity change from 0 to 204800000 [ 3802.340571] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3803.668817] loop0: detected capacity change from 0 to 204800000 [ 3803.677095] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3804.999396] loop0: detected capacity change from 0 to 204800000 [ 3805.011649] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3806.395307] loop0: detected capacity change from 0 to 204800000 [ 3806.419188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3806.430854] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3806.485454] Lustre: 248788:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3806.489989] Lustre: 248788:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 3806.494393] Lustre: 248788:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 3806.499137] Lustre: 248788:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3806.504786] Lustre: 248788:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 3806.511504] Lustre: 248788:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3806.518323] Lustre: 248788:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 3806.521202] Lustre: 248788:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3806.524152] Lustre: 248788:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 3806.526858] Lustre: 248788:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 3806.529737] Lustre: 248788:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 3806.532652] Lustre: 248788:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 3806.535763] Lustre: 248788:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 3806.539970] Lustre: 248788:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 3809.099829] Lustre: server umount MGS complete [ 3809.391901] loop0: detected capacity change from 0 to 204800000 [ 3809.417425] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3809.592662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3809.609860] Lustre: lustre-MDT0000: Aborting client recovery [ 3809.926883] loop1: detected capacity change from 0 to 204800000 [ 3809.957623] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3810.001415] Lustre: 249219:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3810.007795] Lustre: 249219:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3810.012708] Lustre: 249219:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 3810.017338] Lustre: 249219:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 3810.023143] Lustre: 249219:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 3810.028668] Lustre: 249219:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3810.034200] Lustre: 249219:0:(osd_scrub.c:1831:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3810.044914] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3810.118613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3810.148456] Lustre: lustre-MDT0001: Aborting client recovery [ 3810.462951] loop2: detected capacity change from 0 to 204800000 [ 3810.512929] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3810.566277] Lustre: 249300:0:(osd_scrub.c:1831:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3810.569846] Lustre: 249300:0:(osd_scrub.c:1831:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3810.573901] Lustre: 249300:0:(osd_scrub.c:1831:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3810.577582] Lustre: 249300:0:(osd_scrub.c:1831:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3810.581584] Lustre: 249300:0:(osd_scrub.c:1831:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3810.584818] Lustre: 249300:0:(osd_scrub.c:1831:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3810.588334] Lustre: 249300:0:(osd_scrub.c:1831:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3810.642547] Lustre: Found index 0 for lustre-OST0000, updating log [ 3810.680118] Lustre: lustre-OST0000: Aborting recovery [ 3811.005086] loop3: detected capacity change from 0 to 204800000 [ 3811.035419] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3811.082069] Lustre: 249383:0:(osd_scrub.c:1831:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3811.086914] Lustre: 249383:0:(osd_scrub.c:1831:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3811.090328] Lustre: 249383:0:(osd_scrub.c:1831:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3811.093818] Lustre: 249383:0:(osd_scrub.c:1831:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3811.097401] Lustre: 249383:0:(osd_scrub.c:1831:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3811.101231] Lustre: 249383:0:(osd_scrub.c:1831:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3811.105213] Lustre: 249383:0:(osd_scrub.c:1831:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3811.146282] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3811.147975] Lustre: Skipped 1 previous similar message [ 3816.433608] LustreError: 249163:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -108 [ 3820.519292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 3820.521057] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 3820.522577] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 3820.523588] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 3820.554792] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 3820.562226] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 3821.831358] LustreError: 249128:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 515us: rc = -2 [ 3835.874049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3835.875072] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3835.875434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3835.881460] Lustre: Skipped 1 previous similar message [ 3839.414433] Lustre: server umount lustre-MDT0000 complete [ 3839.819057] LustreError: 249111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773671914 with bad export cookie 12702836071272550194 [ 3839.820901] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3839.824585] LustreError: 249111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3840.992066] LustreError: 249397:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3840.992661] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3840.993312] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3840.993320] Lustre: Skipped 3 previous similar messages [ 3841.003745] LustreError: 249397:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 3841.011949] Lustre: Skipped 3 previous similar messages [ 3845.023682] LustreError: 249787:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3845.023884] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3845.029580] LustreError: 249787:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3845.038183] Lustre: Skipped 2 previous similar messages [ 3850.143989] LustreError: 249127:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3850.143996] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3850.150699] LustreError: 249127:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3850.153238] Lustre: Skipped 1 previous similar message [ 3854.303183] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3854.436641] Lustre: server umount lustre-MDT0001 complete [ 3869.151153] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3869.204483] Lustre: server umount lustre-OST0000 complete [ 3875.838546] Lustre: server umount lustre-OST0001 complete [ 3879.603875] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 10:39:13 (1773671953) [ 3891.517522] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 3894.591184] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3910.489708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3910.605307] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3910.622784] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3910.663743] Lustre: lustre-MDT0000: new disk, initializing [ 3910.701906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3910.710344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3912.479477] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3917.965972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3918.027658] Lustre: 253430:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3918.059462] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3918.062467] Lustre: Skipped 1 previous similar message [ 3918.111481] Lustre: lustre-MDT0001: new disk, initializing [ 3918.163409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3918.185327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3918.188671] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3919.985519] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3923.958894] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3925.582466] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3928.479563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3928.617261] Lustre: lustre-OST0000: new disk, initializing [ 3928.619841] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3928.654622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3931.019319] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3934.265224] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3937.788783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3937.791118] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3937.819616] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3941.854834] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 3947.483986] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 3957.993424] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 3967.399991] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 4019.682430] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4019.697066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4019.713223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4019.716989] Lustre: Skipped 1 previous similar message [ 4021.759780] Lustre: server umount lustre-OST0000 complete [ 4029.922466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4029.938399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4029.938805] Lustre: Skipped 1 previous similar message [ 4029.956776] Lustre: Skipped 2 previous similar messages [ 4033.999167] Lustre: server umount lustre-MDT0000 complete [ 4035.041247] LustreError: 254701:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4035.075466] LustreError: 254701:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4039.496761] LustreError: 255916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773672113 with bad export cookie 12702836071272552910 [ 4039.514067] LustreError: 255916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4039.515611] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4040.512136] Lustre: server umount lustre-MDT0001 complete [ 4052.859849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4057.722333] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4062.774132] Lustre: server umount MGS complete [ 4079.378934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4080.230383] LustreError: 259059:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4080.276488] LustreError: 259072:0:(qmt_pool.c:1614:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 4080.402144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4086.842956] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4089.837100] LustreError: 259060:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4089.861877] LustreError: 259060:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4098.355229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4105.701659] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4114.432145] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4118.283714] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4125.901351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4126.224624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4126.229957] Lustre: Skipped 1 previous similar message [ 4132.331597] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4138.558618] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4184.545806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4184.552732] LustreError: Skipped 1 previous similar message [ 4184.554439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4184.566429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4189.790660] Lustre: server umount lustre-OST0000 complete [ 4195.818265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4195.831642] Lustre: Skipped 2 previous similar messages [ 4200.274221] Lustre: server umount lustre-MDT0000 complete [ 4200.928188] LustreError: 261165:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4200.954343] LustreError: 261165:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4204.216221] LustreError: 259783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773672278 with bad export cookie 12702836071272554835 [ 4204.225362] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4204.230228] LustreError: 259783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4204.683296] Lustre: server umount lustre-MDT0001 complete [ 4217.124064] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 4220.281593] Key type lgssc unregistered [ 4220.625151] LNet: 262677:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4220.628321] LNetError: 262677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4220.648988] LNet: Removed LNI 192.168.203.128@tcp [ 4221.616667] Key type .llcrypt unregistered [ 4221.618194] Key type ._llcrypt unregistered [ 4263.325662] Key type ._llcrypt registered [ 4263.330550] Key type .llcrypt registered [ 4263.494972] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 4279.490683] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 4280.822860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4281.032851] alg: No test for adler32 (adler32-zlib) [ 4282.328386] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 4282.565223] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 4284.258676] Key type lgssc registered [ 4285.762541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4324.425993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4324.454650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4325.908422] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4325.955400] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4326.045739] Lustre: lustre-MDT0000: new disk, initializing [ 4326.152679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4326.177610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4331.163986] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4342.948093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4343.114169] Lustre: 266620:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4343.198146] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4343.200713] Lustre: Skipped 1 previous similar message [ 4343.356712] Lustre: lustre-MDT0001: new disk, initializing [ 4343.510296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4343.569914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4343.584708] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4348.861714] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4358.185849] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4361.961202] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4371.045850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4371.357352] Lustre: lustre-OST0000: new disk, initializing [ 4371.360429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4371.457119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4379.049145] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4380.832930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4380.846428] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4381.026506] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4387.970436] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4396.500500] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 4411.145154] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 4413.538355] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 4421.749555] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 4473.313370] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4473.321209] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4473.332512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4473.383455] Lustre: Skipped 1 previous similar message [ 4476.658890] Lustre: server umount lustre-OST0000 complete [ 4482.021587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4482.037094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4482.049532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4482.059812] Lustre: Skipped 1 previous similar message [ 4483.553967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4483.556280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4483.591914] Lustre: Skipped 1 previous similar message [ 4487.750682] Lustre: server umount lustre-MDT0000 complete [ 4491.821661] LustreError: 266612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773672566 with bad export cookie 9695810094559058469 [ 4491.834754] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4491.840392] LustreError: 266612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4492.378819] Lustre: server umount lustre-MDT0001 complete [ 4502.222873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4507.672696] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4513.760804] Lustre: server umount MGS complete [ 4526.985753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4527.760337] LustreError: 272061:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4527.775377] LustreError: 272061:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 4527.789271] LustreError: 272074:0:(qmt_pool.c:1614:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 4527.876193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4533.217960] LustreError: 272062:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4533.287832] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4538.349341] LustreError: 272061:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4543.311122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4543.467204] LustreError: 272062:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4543.939256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4549.323029] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4556.249780] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4561.547857] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4569.776285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4570.256674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4577.147462] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4583.593836] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4609.506702] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4609.514778] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4609.537154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4613.842850] Lustre: server umount lustre-OST0000 complete [ 4620.768807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4620.782659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4620.785136] Lustre: Skipped 2 previous similar messages [ 4620.807044] Lustre: Skipped 3 previous similar messages [ 4624.585658] Lustre: server umount lustre-MDT0000 complete [ 4625.896264] LustreError: 272056:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4625.910582] LustreError: 272056:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4629.006412] LustreError: 272042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773672703 with bad export cookie 9695810094559060394 [ 4629.012386] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4629.014263] LustreError: 272042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4629.319663] Lustre: server umount lustre-MDT0001 complete [ 4642.174110] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 4645.388076] Key type lgssc unregistered [ 4645.753745] LNet: 275653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4645.759860] LNetError: 275653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4646.827597] LNet: Removed LNI 192.168.203.128@tcp [ 4648.128436] Key type .llcrypt unregistered [ 4648.129763] Key type ._llcrypt unregistered [ 4672.297537] Key type ._llcrypt registered [ 4672.300122] Key type .llcrypt registered [ 4672.421422] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 4686.713527] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 4688.215958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4688.251924] alg: No test for adler32 (adler32-zlib) [ 4689.300681] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 4689.686163] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 4691.359616] Key type lgssc registered [ 4692.645115] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4718.623365] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 4734.670083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4734.701400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4736.055629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4736.090457] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4736.177635] Lustre: lustre-MDT0000: new disk, initializing [ 4736.302737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4736.315818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4741.641264] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4752.347692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4752.585448] Lustre: lustre-OST0000: new disk, initializing [ 4752.589595] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4752.596538] Lustre: Skipped 1 previous similar message [ 4752.658383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4758.954092] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4760.161949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 4760.167547] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 4760.201797] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 5656.034272] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5656.051631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5661.154190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5662.017870] Lustre: server umount lustre-MDT0000 complete [ 5666.380391] LustreError: 278841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773673740 with bad export cookie 9232677002746750633 [ 5666.389244] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5666.400201] LustreError: 278841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5666.635246] Lustre: server umount lustre-OST0000 complete [ 5694.719803] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 5705.995781] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 5742.175834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5742.376671] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5742.399150] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5742.459936] Lustre: lustre-MDT0000: new disk, initializing [ 5742.553248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5742.567846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5747.477622] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5759.216104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5759.417559] Lustre: 283950:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5759.500548] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5759.502908] Lustre: Skipped 1 previous similar message [ 5759.622806] Lustre: lustre-MDT0001: new disk, initializing [ 5759.816060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5759.913960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5759.933163] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5765.845570] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5774.758608] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5778.807465] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5786.149157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5786.517212] Lustre: lustre-OST0000: new disk, initializing [ 5786.520944] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5786.620101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5792.906248] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5793.407530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5793.414948] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5793.583101] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5800.884742] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5805.860747] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5806.111549] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5810.868818] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5811.154517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5814.240591] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5814.243880] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5814.256245] LustreError: Skipped 1 previous similar message [ 5814.256959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5814.282269] Lustre: Skipped 1 previous similar message [ 5819.718902] Lustre: server umount lustre-OST0000 complete [ 5824.483068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5824.493991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5824.498779] Lustre: Skipped 1 previous similar message [ 5824.517217] Lustre: Skipped 2 previous similar messages [ 5829.600972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5829.611824] Lustre: Skipped 1 previous similar message [ 5838.305304] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5838.765401] Lustre: server umount lustre-MDT0000 complete [ 5839.840199] LustreError: 283961:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5839.858680] LustreError: 283961:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 5842.666961] LustreError: 283941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773673917 with bad export cookie 9232677002749691445 [ 5842.676732] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5842.684886] LustreError: 283941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5842.977711] Lustre: server umount lustre-MDT0001 complete [ 5846.867453] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [ 5848.816192] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 11:12:01 (1773673921) [ 5855.588935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5856.052345] LustreError: 287857:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5856.116597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5860.714171] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5861.347327] LustreError: 287858:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5866.469550] LustreError: 287857:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5868.362418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5868.690051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5873.436626] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5878.973324] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5882.683364] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5890.060925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5890.590753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5897.570874] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5904.295150] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5912.261355] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5912.465960] Lustre: lustre-OST0001: new disk, initializing [ 5912.469444] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5912.577741] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5919.263923] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5921.926199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5921.948071] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5922.002843] LustreError: 290803:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 5923.051639] LustreError: 290803:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 5923.073064] LustreError: 290803:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 5924.158517] LustreError: 290799:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 5924.178388] LustreError: 290799:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 5926.241384] LustreError: 290799:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 5926.250566] LustreError: 290799:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 5929.338897] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5929.605534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5930.425774] LustreError: 290803:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 5930.440966] LustreError: 290803:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 6 previous similar messages [ 5933.879112] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9f379849f000.ost_server_uuid 50 [ 5935.513907] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9f379849f000.ost_server_uuid in FULL state after 0 sec [ 5938.722535] LustreError: 290799:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 5938.739384] LustreError: 290799:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 16 previous similar messages [ 5939.610451] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f379849f000.ost_server_uuid 50 [ 5941.073830] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9f379849f000.ost_server_uuid in FULL state after 0 sec [ 5947.918540] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5958.136094] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5958.136342] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5958.148912] Lustre: Skipped 1 previous similar message [ 5958.158112] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5958.164560] Lustre: Skipped 3 previous similar messages [ 5962.645629] Lustre: server umount lustre-OST0001 complete [ 5963.244859] LustreError: 289452:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5963.261906] LustreError: 289452:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5970.400312] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5970.402540] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5972.451493] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5973.475918] LustreError: 289452:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5973.499153] LustreError: 289452:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 5974.878112] Lustre: server umount lustre-OST0000 complete [ 5981.164175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5981.174475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5981.199602] Lustre: Skipped 1 previous similar message [ 5981.212569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5981.222632] Lustre: Skipped 5 previous similar messages [ 5985.353358] Lustre: server umount lustre-MDT0000 complete [ 5989.439435] LustreError: 287837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773674063 with bad export cookie 9232677002749692327 [ 5989.441755] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5989.453897] LustreError: 287837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5989.862277] Lustre: server umount lustre-MDT0001 complete [ 6000.841322] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 6004.054872] Key type lgssc unregistered [ 6004.501877] LNet: 292801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6004.524172] LNetError: 292801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6005.548223] LNet: Removed LNI 192.168.203.128@tcp [ 6006.416907] Key type .llcrypt unregistered [ 6006.418878] Key type ._llcrypt unregistered [ 6031.156598] Key type ._llcrypt registered [ 6031.158948] Key type .llcrypt registered [ 6031.229675] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 6032.144634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6032.186412] alg: No test for adler32 (adler32-zlib) [ 6033.384837] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 6033.607871] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 6035.312957] Key type lgssc registered [ 6036.415361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6043.878825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6043.903502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6045.458621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6050.455587] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6054.885266] LustreError: 294209:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6054.904293] LustreError: 294209:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 6055.962568] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6060.004089] LustreError: 294208:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6063.559186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6063.990394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6067.860194] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6078.620571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6079.027329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6085.791382] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6089.250353] LustreError: 295757:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6090.294961] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 6091.654210] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6095.387461] LustreError: 295757:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6095.410089] LustreError: 295757:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 6098.929147] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6099.298158] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6104.591315] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6105.998671] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6111.656628] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6119.463130] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6119.698214] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6227.442585] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6227.443072] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6227.484937] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6227.487322] Lustre: Skipped 1 previous similar message [ 6231.822757] Lustre: server umount lustre-OST0001 complete [ 6232.550306] LustreError: 297529:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6232.573133] LustreError: 297529:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 6238.689752] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6238.697073] LustreError: Skipped 1 previous similar message [ 6238.700076] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6238.713959] Lustre: Skipped 1 previous similar message [ 6238.719085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6242.788439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6242.794749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6242.801479] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6242.806061] LustreError: 295757:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6242.806073] LustreError: 295757:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 6242.866737] Lustre: Skipped 1 previous similar message [ 6244.747335] Lustre: server umount lustre-OST0000 complete [ 6253.025856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6253.028960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6253.043555] Lustre: Skipped 1 previous similar message [ 6255.814277] Lustre: server umount lustre-MDT0000 complete [ 6260.190197] LustreError: 294188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773674334 with bad export cookie 1132865403860046836 [ 6260.196241] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6260.198468] LustreError: 294188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6260.700291] Lustre: server umount lustre-MDT0001 complete [ 6274.971431] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 6278.557937] Key type lgssc unregistered [ 6278.956282] LNet: 298768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6278.971091] LNetError: 298768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6278.992438] LNet: Removed LNI 192.168.203.128@tcp [ 6279.957115] Key type .llcrypt unregistered [ 6279.960848] Key type ._llcrypt unregistered [ 6327.052540] Key type ._llcrypt registered [ 6327.054496] Key type .llcrypt registered [ 6327.147875] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 6328.256352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6328.466836] alg: No test for adler32 (adler32-zlib) [ 6329.804390] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 6330.146628] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 6332.009097] Key type lgssc registered [ 6334.133306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6348.575688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6348.621770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6350.042459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6354.484646] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6360.563246] LustreError: 301045:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6360.580668] LustreError: 301045:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 6364.394632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6364.844425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6370.517460] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6374.415938] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6381.815905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6382.194461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6389.653104] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6390.372818] LustreError: 302504:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6395.899202] LustreError: 302503:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6396.977651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 6399.127337] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6399.588647] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6404.633868] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 6406.333313] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6415.082816] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6422.678736] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6440.421047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6440.424547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6440.435732] Lustre: Skipped 2 previous similar messages [ 6440.449517] Lustre: Skipped 2 previous similar messages [ 6445.363439] Lustre: server umount lustre-MDT0000 complete [ 6445.539741] LustreError: 301976:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6445.562562] LustreError: 301976:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 5 previous similar messages [ 6448.850248] LustreError: 301025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773674523 with bad export cookie 10946702031315916674 [ 6448.857613] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6448.876172] LustreError: 301025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6449.178423] Lustre: server umount lustre-MDT0001 complete [ 6463.216522] Lustre: server umount lustre-OST0000 complete [ 6477.218968] Lustre: server umount lustre-OST0001 complete [ 6493.318197] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 6502.041184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6502.406358] LustreError: 306235:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6502.487582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6505.889537] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6511.652629] LustreError: 306236:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6511.677289] LustreError: 306236:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 6515.666789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6516.278266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6520.761779] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6524.248918] Lustre: 307341:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6530.850578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6536.531588] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6541.357345] LustreError: 308266:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6545.012870] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6545.309969] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6545.321787] Lustre: Skipped 1 previous similar message [ 6548.279693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 6548.290907] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 6550.265749] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6557.863737] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6561.685192] Lustre: 309184:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6572.513235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6572.524227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6572.533442] Lustre: Skipped 1 previous similar message [ 6572.539897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6574.051951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6574.052622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6574.061795] Lustre: Skipped 2 previous similar messages [ 6574.067511] Lustre: Skipped 2 previous similar messages [ 6577.342853] Lustre: server umount lustre-MDT0000 complete [ 6579.176790] LustreError: 306236:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6579.207414] LustreError: 306236:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 6580.945397] LustreError: 306217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773674655 with bad export cookie 10946702031315918375 [ 6580.947461] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6580.952815] LustreError: 306217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6581.432207] Lustre: server umount lustre-MDT0001 complete [ 6594.788691] Lustre: server umount lustre-OST0000 complete [ 6608.405499] Lustre: server umount lustre-OST0001 complete [ 6612.066347] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [ 6613.569473] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [ 6615.287744] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 11:24:47 (1773674687) [ 6635.852910] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 6644.012467] loop0: detected capacity change from 0 to 1099511627776 [ 6644.467965] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 6644.471688] *** VALIDATE xfs *** [ 6644.476547] XFS (loop0): Mounting V5 Filesystem [ 6644.497360] XFS (loop0): Ending clean mount [ 6645.812266] loop1: detected capacity change from 0 to 18691697672192 [ 6745.354276] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 6752.680202] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 6785.409492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6785.655978] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6785.692882] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6785.761619] Lustre: lustre-MDT0000: new disk, initializing [ 6785.856431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6785.890405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6789.507727] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6800.150248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6800.262158] Lustre: 314270:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6800.318340] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6800.321394] Lustre: Skipped 1 previous similar message [ 6800.409836] Lustre: lustre-MDT0001: new disk, initializing [ 6800.515207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6800.530697] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6806.142162] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6814.213527] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6818.971654] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6825.824194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6826.155466] Lustre: lustre-OST0000: new disk, initializing [ 6826.170220] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6828.115722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6828.132898] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6828.296386] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6832.594990] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6841.634906] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6847.094254] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6847.438564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6852.928426] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6853.228513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6859.235903] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6859.239756] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6859.244249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6859.246996] Lustre: Skipped 1 previous similar message [ 6859.253704] Lustre: Skipped 2 previous similar messages [ 6861.509296] Lustre: server umount lustre-OST0000 complete [ 6866.920830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6866.942248] LustreError: Skipped 1 previous similar message [ 6866.948077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6866.971074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6878.693075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6878.704023] Lustre: Skipped 5 previous similar messages [ 6879.199123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6879.866375] Lustre: server umount lustre-MDT0000 complete [ 6883.818232] LustreError: 317147:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6883.855892] LustreError: 317147:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 6884.554397] LustreError: 314261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773674958 with bad export cookie 10946702031315919922 [ 6884.560980] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6884.564692] LustreError: 314261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6885.186725] Lustre: server umount lustre-MDT0001 complete [ 6894.283354] XFS (loop0): Unmounting Filesystem [ 6898.562438] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 11:29:31 (1773674971) [ 6906.240420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6906.682362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6906.685094] Lustre: Skipped 2 previous similar messages [ 6911.316167] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6919.373521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6924.634236] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6929.627651] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6932.837311] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6939.412805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6945.063745] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6950.439377] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6965.728382] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6965.740273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6965.752965] Lustre: Skipped 1 previous similar message [ 6965.757702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6969.400295] Lustre: server umount lustre-OST0000 complete [ 6979.066520] Lustre: server umount lustre-MDT0000 complete [ 6982.851396] LustreError: 320962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773675057 with bad export cookie 10946702031315920811 [ 6982.855148] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6982.859962] LustreError: 320962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6983.264734] Lustre: server umount lustre-MDT0001 complete [ 6993.397126] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 6995.992481] Key type lgssc unregistered [ 6996.199480] LNet: 321698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6996.205193] LNetError: 321698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6996.227794] LNet: Removed LNI 192.168.203.128@tcp [ 6997.012716] Key type .llcrypt unregistered [ 6997.016155] Key type ._llcrypt unregistered [ 7022.233949] Key type ._llcrypt registered [ 7022.235587] Key type .llcrypt registered [ 7022.343487] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 7023.614380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7023.709446] alg: No test for adler32 (adler32-zlib) [ 7024.771532] Lustre: Lustre: Build Version: 2.17.51_1_gb548ff5 [ 7024.938056] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 7026.615608] Key type lgssc registered [ 7027.753569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7035.532287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7035.550949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7037.326200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7042.537317] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7046.630837] LustreError: 323251:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7046.659640] LustreError: 323251:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 7051.750354] LustreError: 323252:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7051.849446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7052.317926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7057.258726] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7063.867795] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7068.509934] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7075.286620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7075.703578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7082.433653] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7088.957201] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7097.315312] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7097.328994] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7101.753075] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.28@tcp (stopping) [ 7101.756974] Lustre: Skipped 1 previous similar message [ 7103.693419] Lustre: server umount lustre-MDT0001 complete [ 7106.837673] LustreError: 323246:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7111.972521] LustreError: 324459:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7112.002435] LustreError: 324459:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 7114.093403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7114.189844] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7114.194417] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7114.231959] Lustre: 323241:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7114.523947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7118.942034] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7119.849782] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7122.940823] LustreError: 326502:0:(obd_mount.c:212:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 7122.947108] LustreError: 326502:0:(tgt_mount.c:667:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 7122.952762] LustreError: 323244:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 7123.018502] LustreError: 326505:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.203.128@tcp: cfg command failed: rc = -17 [ 7123.034796] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 7123.034796] [ 7427.042855] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7427.047063] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7427.062526] Lustre: Skipped 1 previous similar message [ 7427.068027] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7427.070484] Lustre: Skipped 2 previous similar messages [ 7429.456208] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.28@tcp (stopping) [ 7434.520497] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.28@tcp (stopping) [ 7434.528386] Lustre: Skipped 2 previous similar messages [ 7440.353449] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7440.601397] Lustre: server umount lustre-MDT0001 complete [ 7442.408504] LustreError: 323248:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7442.425021] LustreError: 323248:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 7452.186623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7452.295943] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7452.298522] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7452.372298] Lustre: 323238:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7452.666457] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 7457.256635] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7457.786983] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7458.829396] LustreError: 334930:0:(obd_mount.c:212:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 7458.833373] LustreError: 334930:0:(tgt_mount.c:667:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 7458.842714] LustreError: 323244:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 7458.846322] LustreError: 323244:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 7458.918305] LustreError: 334936:0:(obd_config.c:797:class_setup()) Device 8 already setup (type osp) [ 7458.924169] LustreError: 334936:0:(obd_config.c:902:class_detach()) OBD device 8 still set up [ 7458.933221] LustreError: 334936:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.203.128@tcp: cfg command failed: rc = -17 [ 7458.947697] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.203.128@tcp [ 7458.947697] [ 7764.967094] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7764.970970] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7765.007721] Lustre: Skipped 1 previous similar message [ 7765.034543] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7765.034551] Lustre: Skipped 3 previous similar messages [ 7769.061593] Lustre: server umount lustre-MDT0001 complete [ 7770.084488] LustreError: 338799:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7770.099783] LustreError: 338799:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 5 previous similar messages [ 7778.417321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7778.503914] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7778.512108] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7778.571882] Lustre: 323241:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7778.842130] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 7783.027517] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7783.914660] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7785.976265] LustreError: 343668:0:(obd_mount.c:212:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 7785.987947] LustreError: 343668:0:(tgt_mount.c:667:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 7786.003795] LustreError: 323244:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 7786.006035] LustreError: 323244:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 7786.050396] LustreError: 343671:0:(lod_lov.c:212:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 7786.063392] LustreError: 343671:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.203.128@tcp: cfg command failed: rc = -114 [ 7786.073261] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 7786.073261] [ 8104.419292] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8104.419988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8104.428270] Lustre: Skipped 1 previous similar message [ 8104.433194] Lustre: Skipped 2 previous similar messages [ 8116.191371] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8116.342343] Lustre: server umount lustre-MDT0000 complete [ 8119.423053] LustreError: 326102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773676193 with bad export cookie 15303032719702580541 [ 8119.430210] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8119.437862] LustreError: 326102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8119.776154] LustreError: 329233:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8119.777143] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8119.793895] LustreError: 329233:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 8119.803997] Lustre: Skipped 1 previous similar message [ 8125.818474] Lustre: server umount lustre-MDT0001 complete [ 8129.130317] Lustre: server umount lustre-OST0000 complete [ 8136.790693] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 8144.080525] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 8175.529918] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 11:50:48 (1773676248) [ 8180.785703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8180.996777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8181.034439] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8181.105295] Lustre: lustre-MDT0000: new disk, initializing [ 8181.184285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8181.202269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8184.540515] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8194.279908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8194.476965] Lustre: 354911:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8194.577794] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8194.580847] Lustre: Skipped 1 previous similar message [ 8194.683222] Lustre: lustre-MDT0001: new disk, initializing [ 8194.844326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8194.881911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8194.892028] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8199.332785] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8206.690284] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8210.322061] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8216.967090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8217.265494] Lustre: lustre-OST0000: new disk, initializing [ 8217.269814] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8217.362217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8223.351898] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 8224.860102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8224.873216] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8225.014257] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8230.401286] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8239.076835] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 8239.084522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8239.095345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8239.103964] Lustre: Skipped 10 previous similar messages [ 8242.503710] Lustre: server umount lustre-MDT0000 complete [ 8245.544563] LustreError: 354905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773676319 with bad export cookie 15303032719702582494 [ 8245.547170] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8245.555635] LustreError: 354905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8245.574760] LustreError: 355702:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [ 8245.735609] LustreError: 354921:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8245.746115] LustreError: 354921:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 8245.910818] Lustre: server umount lustre-MDT0001 complete [ 8251.612396] Lustre: server umount lustre-OST0000 complete [ 8276.683655] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 8284.736801] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 8311.814198] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8179 sec ========== 11:53:04 (1773676384) [ 8313.546689] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:53:06 (1773676386) === [ 8316.600393] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:53:09 (1773676389) === [ 8352.527818] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 8354.740794] Key type lgssc unregistered [ 8355.063311] LNet: 362855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8355.067165] LNetError: 362855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8356.134090] LNet: Removed LNI 192.168.203.128@tcp [ 8356.890699] Key type .llcrypt unregistered [ 8356.892944] Key type ._llcrypt unregistered