[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 442948222 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003190] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.007877] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.010153] LSM: Security Framework initializing [ 0.011041] Yama: becoming mindful. [ 0.012032] SELinux: Initializing. [ 0.013055] *** VALIDATE selinux *** [ 0.021256] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026303] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027144] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028092] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029122] *** VALIDATE tmpfs *** [ 0.031095] *** VALIDATE proc *** [ 0.032228] *** VALIDATE cgroup *** [ 0.033005] *** VALIDATE cgroup2 *** [ 0.034231] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037026] Spectre V2 : User space: Vulnerable [ 0.038004] Speculative Store Bypass: Vulnerable [ 0.041747] debug: unmapping init [mem 0xffffffffad459000-0xffffffffad460fff] [ 0.043924] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044711] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045019] ... version: 2 [ 0.046007] ... bit width: 48 [ 0.047006] ... generic registers: 4 [ 0.048007] ... value mask: 0000ffffffffffff [ 0.049009] ... max period: 00007fffffffffff [ 0.050013] ... fixed-purpose events: 3 [ 0.051007] ... event mask: 000000070000000f [ 0.053261] rcu: Hierarchical SRCU implementation. [ 0.055567] smp: Bringing up secondary CPUs ... [ 0.056485] x86: Booting SMP configuration: [ 0.057014] .... node #0, CPUs: #1 #2 #3 [ 0.066020] smp: Brought up 1 node, 4 CPUs [ 0.068014] smpboot: Max logical packages: 1 [ 0.069015] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.195268] node 0 deferred pages initialised in 125ms [ 0.198142] devtmpfs: initialized [ 0.199199] x86/mm: Memory block size: 128MB [ 0.201727] gcov: version magic: 0x41383552 [ 0.205459] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.208100] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.210391] pinctrl core: initialized pinctrl subsystem [ 0.212201] [ 0.212782] ************************************************************* [ 0.215011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.218018] ** ** [ 0.223024] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.225015] ** ** [ 0.227014] ** This means that this kernel is built to expose internal ** [ 0.230015] ** IOMMU data structures, which may compromise security on ** [ 0.232015] ** your system. ** [ 0.237024] ** ** [ 0.239114] ** If you see this message and you are not debugging the ** [ 0.242013] ** kernel, report this immediately to your vendor! ** [ 0.244015] ** ** [ 0.246014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.248020] ************************************************************* [ 0.250693] NET: Registered protocol family 16 [ 0.252458] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.255144] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.257047] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.260146] cpuidle: using governor menu [ 0.261806] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.265618] PCI: Using configuration type 1 for base access [ 0.268159] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.278124] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.280046] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.283188] cryptd: max_cpu_qlen set to 1000 [ 0.286246] ACPI: Added _OSI(Module Device) [ 0.287014] ACPI: Added _OSI(Processor Device) [ 0.289011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.291010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.296798] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.305526] ACPI: Interpreter enabled [ 0.307063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.308008] ACPI: Using IOAPIC for interrupt routing [ 0.310161] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.314450] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.327794] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.330032] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.333077] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.337183] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.344238] acpiphp: Slot [2] registered [ 0.346056] acpiphp: Slot [5] registered [ 0.347000] acpiphp: Slot [6] registered [ 0.347103] acpiphp: Slot [7] registered [ 0.349183] acpiphp: Slot [8] registered [ 0.350111] acpiphp: Slot [9] registered [ 0.351217] acpiphp: Slot [10] registered [ 0.352084] acpiphp: Slot [3] registered [ 0.353000] acpiphp: Slot [4] registered [ 0.353000] acpiphp: Slot [11] registered [ 0.354061] acpiphp: Slot [12] registered [ 0.355075] acpiphp: Slot [13] registered [ 0.357093] acpiphp: Slot [14] registered [ 0.358060] acpiphp: Slot [15] registered [ 0.359092] acpiphp: Slot [16] registered [ 0.361071] acpiphp: Slot [17] registered [ 0.362135] acpiphp: Slot [18] registered [ 0.363074] acpiphp: Slot [19] registered [ 0.365084] acpiphp: Slot [20] registered [ 0.366059] acpiphp: Slot [21] registered [ 0.367062] acpiphp: Slot [22] registered [ 0.369185] acpiphp: Slot [23] registered [ 0.370059] acpiphp: Slot [24] registered [ 0.371078] acpiphp: Slot [25] registered [ 0.373120] acpiphp: Slot [26] registered [ 0.374260] acpiphp: Slot [27] registered [ 0.376086] acpiphp: Slot [28] registered [ 0.377076] acpiphp: Slot [29] registered [ 0.379091] acpiphp: Slot [30] registered [ 0.380174] acpiphp: Slot [31] registered [ 0.382052] PCI host bridge to bus 0000:00 [ 0.383016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.386039] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.388020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.390014] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.393019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.396023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.397171] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.399841] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.402045] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.412013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.416891] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.419009] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.421011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.423011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.425733] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.428779] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.431040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.433755] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.439016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.451743] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.457015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.461697] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.474022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.481109] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.499020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.510000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.522020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.528013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.556024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.570815] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.576019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.582016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.594020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.603518] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.613020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.620023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.641026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.651894] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.657022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.665022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.680023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.686647] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.691012] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.695015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.705019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.715985] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.718305] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.721478] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.723333] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.725170] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.730098] iommu: Default domain type: Passthrough [ 0.731000] SCSI subsystem initialized [ 0.731000] ACPI: bus type USB registered [ 0.731000] usbcore: registered new interface driver usbfs [ 0.734059] usbcore: registered new interface driver hub [ 0.736069] usbcore: registered new device driver usb [ 0.738179] pps_core: LinuxPPS API ver. 1 registered [ 0.739007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.743083] PTP clock support registered [ 0.745057] EDAC MC: Ver: 3.0.0 [ 0.746482] PCI: Using ACPI for IRQ routing [ 0.747793] NetLabel: Initializing [ 0.748006] NetLabel: domain hash size = 128 [ 0.749008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.750140] NetLabel: unlabeled traffic allowed by default [ 0.752162] vgaarb: loaded [ 0.754390] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.756016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.763287] clocksource: Switched to clocksource kvm-clock [ 0.870378] VFS: Disk quotas dquot_6.6.0 [ 0.871184] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.872674] *** VALIDATE ramfs *** [ 0.873862] *** VALIDATE hugetlbfs *** [ 0.875990] pnp: PnP ACPI init [ 0.878384] pnp: PnP ACPI: found 6 devices [ 0.893296] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.896245] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.899141] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.901189] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.903673] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.906442] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.909716] NET: Registered protocol family 2 [ 0.911639] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.916153] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.920074] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.925936] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.929449] TCP: Hash tables configured (established 65536 bind 65536) [ 0.932460] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.935655] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.937951] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.940087] NET: Registered protocol family 1 [ 0.942900] RPC: Registered named UNIX socket transport module. [ 0.944937] RPC: Registered udp transport module. [ 0.946250] RPC: Registered tcp transport module. [ 0.947560] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.949498] NET: Registered protocol family 44 [ 0.950737] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.952460] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.954307] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.956419] PCI: CLS 0 bytes, default 64 [ 0.958168] Unpacking initramfs... [ 2.531409] debug: unmapping init [mem 0xffff9b6b7cc54000-0xffff9b6b7ffbffff] [ 2.535689] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.538287] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.541465] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.296349] Initialise system trusted keyrings [ 3.300649] Key type blacklist registered [ 3.303465] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.327971] zbud: loaded [ 3.331493] *** VALIDATE nfs *** [ 3.332544] *** VALIDATE nfs4 *** [ 3.334356] pstore: using deflate compression [ 3.338214] Platform Keyring initialized [ 3.459636] NET: Registered protocol family 38 [ 3.461328] Key type asymmetric registered [ 3.462438] Asymmetric key parser 'x509' registered [ 3.463964] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.466520] io scheduler mq-deadline registered [ 3.468651] io scheduler kyber registered [ 3.470180] io scheduler bfq registered [ 3.471601] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.473961] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.477503] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.479721] ACPI: Power Button [PWRF] [ 3.484576] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.490513] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.511547] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.528328] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.566426] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.613222] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.670751] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.684506] Non-volatile memory driver v1.3 [ 3.689359] Linux agpgart interface v0.103 [ 3.788611] virtio_blk virtio1: [vda] 67992 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.796681] vda: detected capacity change from 0 to 34811904 [ 3.834797] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.842565] vdb: detected capacity change from 0 to 1073741824 [ 3.888403] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.894470] vdc: detected capacity change from 0 to 2621440000 [ 3.924814] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.931724] vdd: detected capacity change from 0 to 2621440000 [ 3.966853] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.972877] vde: detected capacity change from 0 to 4294967296 [ 4.007199] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.010182] vdf: detected capacity change from 0 to 4294967296 [ 4.029145] libphy: Fixed MDIO Bus: probed [ 4.042719] usbcore: registered new interface driver usbserial_generic [ 4.044532] usbserial: USB Serial support registered for generic [ 4.046943] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.054081] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.058590] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.063500] mousedev: PS/2 mouse device common for all mice [ 4.066255] rtc_cmos 00:05: RTC can wake from S4 [ 4.071459] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.072514] rtc_cmos 00:05: registered as rtc0 [ 4.078921] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.080250] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.082538] intel_pstate: CPU model not supported [ 4.087692] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.112891] hid: raw HID events driver (C) Jiri Kosina [ 4.114575] usbcore: registered new interface driver usbhid [ 4.117946] usbhid: USB HID core driver [ 4.121256] drop_monitor: Initializing network drop monitor service [ 4.126441] Initializing XFRM netlink socket [ 4.129082] NET: Registered protocol family 10 [ 4.134296] Segment Routing with IPv6 [ 4.136882] NET: Registered protocol family 17 [ 4.139394] mpls_gso: MPLS GSO support [ 4.150962] RAS: Correctable Errors collector initialized. [ 4.155099] AVX version of gcm_enc/dec engaged. [ 4.157381] AES CTR mode by8 optimization enabled [ 4.304150] sched_clock: Marking stable (4304132047, 0)->(5171911018, -867778971) [ 4.307387] registered taskstats version 1 [ 4.309565] Loading compiled-in X.509 certificates [ 4.311155] zswap: loaded using pool lzo/zbud [ 4.333226] Key type big_key registered [ 4.349381] Key type encrypted registered [ 4.350426] ima: No TPM chip found, activating TPM-bypass! [ 4.351734] ima: Allocated hash algorithm: sha1 [ 4.352797] ima: No architecture policies found [ 4.358531] evm: Initialising EVM extended attributes: [ 4.363608] evm: security.selinux [ 4.364701] evm: security.ima [ 4.365669] evm: security.capability [ 4.367160] evm: HMAC attrs: 0x1 [ 4.369532] rtc_cmos 00:05: setting system clock to 2026-02-28 18:51:34 UTC (1772304694) [ 4.376896] debug: unmapping init [mem 0xffffffffae403000-0xffffffffae5fffff] [ 4.380239] debug: unmapping init [mem 0xffffffffad182000-0xffffffffad458fff] [ 4.395109] Write protecting the kernel read-only data: 28672k [ 4.406107] debug: unmapping init [mem 0xffffffffab803000-0xffffffffab9fffff] [ 4.410285] debug: unmapping init [mem 0xffffffffac114000-0xffffffffac1fffff] [ 4.496150] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.511167] systemd[1]: Detected virtualization kvm. [ 4.515202] systemd[1]: Detected architecture x86-64. [ 4.517560] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.563987] systemd[1]: No hostname configured. [ 4.567554] systemd[1]: Set hostname to . [ 4.570195] random: systemd: uninitialized urandom read (16 bytes read) [ 4.574150] systemd[1]: Initializing machine ID from random generator. [ 4.674462] random: ln: uninitialized urandom read (6 bytes read) [ 4.821453] random: systemd: uninitialized urandom read (16 bytes read) [ 4.823843] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.838236] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 4.851623] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.638889] device-mapper: uevent: version 1.0.3 [ 5.640464] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.916055] random: fast init done [ 6.982435] virtio_net virtio0 ens2: renamed from eth0 [ 7.160301] scsi host0: ata_piix [ 7.197090] scsi host1: ata_piix [ 7.198629] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.201331] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.642600] dracut-initqueue[588]: RTNETLINK answers: File exists [ 11.875713] random: crng init done [ 11.877341] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 13.294666] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.895656] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.391155] SELinux: Disabled at runtime. [ 15.467051] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.477459] systemd[1]: Detected virtualization kvm. [ 15.479240] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.092748] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.096511] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.103852] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.107207] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.109407] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.124778] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.131345] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 16.230627] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 16.810891] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.200835] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 17.225175] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 17.355033] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 17.368839] EDAC sbridge: Ver: 1.1.2 [ 20.113824] Key type dns_resolver registered [ 20.695098] NFS: Registering the id_resolver key type [ 20.699966] Key type id_resolver registered [ 20.701135] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg447-server login: [ 40.681902] libcfs: loading out-of-tree module taints kernel. [ 40.694070] alg: No test for adler32 (adler32-zlib) [ 41.445226] Key type ._llcrypt registered [ 41.445940] Key type .llcrypt registered [ 41.471394] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 47.629290] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 48.129379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 48.331800] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 48.566592] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 48.568122] LNet: Accept secure, port 988 [ 50.167074] Key type lgssc registered [ 50.651501] Lustre: Echo OBD driver; http://www.lustre.org/ [ 57.055590] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 58.353466] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 61.933920] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 64.467553] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 66.980889] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 71.777282] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 75.618900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 75.642719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 76.717318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 76.725722] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 76.758019] Lustre: lustre-MDT0000: new disk, initializing [ 76.777348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 76.782821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 77.999231] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 82.673749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 82.699385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 82.727686] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 82.736120] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 82.736568] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 82.737852] Lustre: Skipped 1 previous similar message [ 82.772353] Lustre: lustre-MDT0001: new disk, initializing [ 82.789833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 82.796397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 82.799257] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 84.105047] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 89.048602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 89.069696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 89.072290] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 89.154302] Lustre: lustre-OST0000: new disk, initializing [ 89.155990] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 89.172654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 90.346891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 90.350097] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 90.496084] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 95.609196] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 95.634621] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 95.637093] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 95.668247] Lustre: lustre-OST0001: new disk, initializing [ 95.669862] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 95.684772] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 96.748315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 96.752310] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 97.050612] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 102.558626] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 107.215750] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 113.573787] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing check_logdir /tmp/testlogs/ [ 115.172549] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing yml_node [ 116.497522] Lustre: DEBUG MARKER: Client: 2.15.8.1 [ 117.266183] Lustre: DEBUG MARKER: MDS: 2.15.8.1 [ 118.006114] Lustre: DEBUG MARKER: OSS: 2.15.8.1 [ 118.494359] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Feb 28 13:53:29 EST 2026 [ 120.817262] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball 110 [ 121.282696] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 134.111498] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 134.113768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 134.116933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 137.695446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 137.695947] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 137.697932] Lustre: Skipped 2 previous similar messages [ 137.703646] Lustre: Skipped 2 previous similar messages [ 138.878432] Lustre: server umount lustre-MDT0000 complete [ 139.927734] LustreError: 6215:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772304830 with bad export cookie 4348307532235373938 [ 139.929158] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 139.930963] LustreError: 6215:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 142.816508] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 142.816633] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 142.816876] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 142.816881] Lustre: Skipped 1 previous similar message [ 142.821327] LustreError: Skipped 1 previous similar message [ 142.824865] Lustre: Skipped 1 previous similar message [ 144.864056] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 144.868777] LustreError: Skipped 1 previous similar message [ 145.888075] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 145.895093] LustreError: Skipped 1 previous similar message [ 146.095976] Lustre: server umount lustre-MDT0001 complete [ 153.513353] Lustre: server umount lustre-OST0000 complete [ 160.846365] Lustre: server umount lustre-OST0001 complete [ 162.700715] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 165.096818] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 168.111445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 169.999829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 171.680873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 173.494208] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 175.831391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 175.856707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 175.961562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 175.973294] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 176.008359] Lustre: lustre-MDT0000: new disk, initializing [ 176.034216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 176.042276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 177.613300] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 181.692161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 181.713601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 181.744638] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 181.834971] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 183.253235] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 187.083911] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 188.951435] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 191.233170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 191.257698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 191.261578] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 191.355149] Lustre: lustre-OST0000: new disk, initializing [ 191.356426] Lustre: Skipped 1 previous similar message [ 191.357820] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 191.359984] Lustre: Skipped 2 previous similar messages [ 191.383658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 191.385538] Lustre: Skipped 1 previous similar message [ 192.784682] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 193.143357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 193.145673] Lustre: Skipped 1 previous similar message [ 193.147298] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 196.552680] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 197.941677] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 198.005326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 199.348445] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 199.424608] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 203.745141] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 203.745459] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 203.748941] LustreError: Skipped 1 previous similar message [ 203.753265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 203.756490] Lustre: Skipped 6 previous similar messages [ 206.163915] Lustre: server umount lustre-OST0000 complete [ 208.864471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 208.868660] Lustre: Skipped 2 previous similar messages [ 213.985048] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 213.989579] LustreError: Skipped 1 previous similar message [ 214.714958] LustreError: 14300:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772304904 with bad export cookie 4348307532235375366 [ 214.717876] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 214.718405] LustreError: 14300:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 214.837414] Lustre: server umount lustre-MDT0001 complete [ 214.838756] Lustre: Skipped 1 previous similar message [ 216.106682] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 13:55:06 (1772304906) [ 216.830680] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 217.311425] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 13:55:08 (1772304908) [ 219.570724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 219.732762] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 219.758108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 221.118248] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 223.867653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 225.446018] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 227.924184] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 229.756154] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 231.923617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 233.398537] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 235.939713] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 244.705429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 244.706075] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 244.706889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 244.706894] Lustre: Skipped 2 previous similar messages [ 244.709651] Lustre: Skipped 1 previous similar message [ 248.683288] Lustre: server umount lustre-OST0000 complete [ 254.944441] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 257.286210] LustreError: 18831:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772304947 with bad export cookie 4348307532235376220 [ 257.290058] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 262.235461] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 263.329472] Key type lgssc unregistered [ 263.439298] LNet: 22501:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 263.443257] LNet: Removed LNI 192.168.204.147@tcp [ 263.763296] Key type .llcrypt unregistered [ 263.764361] Key type ._llcrypt unregistered [ 272.287540] alg: No test for adler32 (adler32-zlib) [ 273.039286] Key type ._llcrypt registered [ 273.040325] Key type .llcrypt registered [ 273.075325] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 273.468037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 273.663302] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 273.743846] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 273.745366] LNet: Accept secure, port 988 [ 275.327101] Key type lgssc registered [ 275.656629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 278.339109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 279.499521] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 279.515142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 280.970467] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 283.843638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 283.969974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 285.340662] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 287.849930] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 289.715826] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 291.938873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 292.063619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 293.418198] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 295.143243] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 295.873684] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 304.609348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 304.609543] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 304.610854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 304.616110] Lustre: Skipped 1 previous similar message [ 309.728934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 309.731359] Lustre: Skipped 2 previous similar messages [ 310.686542] Lustre: server umount lustre-OST0000 complete [ 314.849021] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 314.849098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 314.849698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 314.861062] Lustre: Skipped 1 previous similar message [ 318.096702] Lustre: server umount lustre-MDT0000 complete [ 319.352253] LustreError: 23940:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772305009 with bad export cookie 4188826482263715459 [ 319.355589] LustreError: 23940:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 319.355755] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 319.491320] Lustre: server umount lustre-MDT0001 complete [ 323.990864] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 325.039338] Key type lgssc unregistered [ 325.155326] LNet: 27612:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 325.157634] LNet: Removed LNI 192.168.204.147@tcp [ 325.431159] Key type .llcrypt unregistered [ 325.432325] Key type ._llcrypt unregistered [ 333.821070] alg: No test for adler32 (adler32-zlib) [ 334.572405] Key type ._llcrypt registered [ 334.573729] Key type .llcrypt registered [ 334.609338] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 335.047531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 335.189988] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 335.266489] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 335.268429] LNet: Accept secure, port 988 [ 336.855118] Key type lgssc registered [ 337.214739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 340.049047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 341.205173] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 341.220740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 342.919633] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 345.738612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 345.857842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 347.265425] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 349.739662] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 351.590934] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 353.615698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 353.733340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 354.948294] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 357.066598] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 357.859589] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 366.559740] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 366.561856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 366.565106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 367.073346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 367.073506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 367.075158] Lustre: Skipped 1 previous similar message [ 367.079710] Lustre: Skipped 1 previous similar message [ 368.465590] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.47@tcp (stopping) [ 368.468619] Lustre: Skipped 1 previous similar message [ 371.587452] Lustre: server umount lustre-MDT0000 complete [ 372.193940] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 372.198305] LustreError: Skipped 2 previous similar messages [ 372.861269] LustreError: 29055:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772305062 with bad export cookie 40179117770176456 [ 372.863751] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 372.865795] LustreError: 29055:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 373.580530] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 373.588119] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.47@tcp (stopping) [ 377.311872] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 377.312095] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 378.336253] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 378.338309] Lustre: Skipped 1 previous similar message [ 379.073992] Lustre: server umount lustre-MDT0001 complete [ 397.033307] Lustre: server umount lustre-OST0000 complete [ 403.516657] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 404.535455] Key type lgssc unregistered [ 404.661499] LNet: 33056:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 404.666166] LNet: Removed LNI 192.168.204.147@tcp [ 404.976470] Key type .llcrypt unregistered [ 404.977992] Key type ._llcrypt unregistered [ 413.937139] alg: No test for adler32 (adler32-zlib) [ 414.688385] Key type ._llcrypt registered [ 414.689699] Key type .llcrypt registered [ 414.732461] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 415.218656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 415.383396] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 415.468473] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 415.470177] LNet: Accept secure, port 988 [ 417.055145] Key type lgssc registered [ 417.416526] Lustre: Echo OBD driver; http://www.lustre.org/ [ 420.006675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 421.153629] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 421.167986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 422.566713] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 425.365153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 425.489700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 426.942339] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 429.442143] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 431.350973] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 433.620397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 433.734221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 434.941722] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 436.835835] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 437.107515] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 445.409267] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 445.409547] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 445.412621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 445.416397] Lustre: Skipped 1 previous similar message [ 445.419304] Lustre: Skipped 1 previous similar message [ 448.850591] Lustre: lustre-OST0000: Not available for connect from 192.168.204.47@tcp (stopping) [ 450.526846] Lustre: server umount lustre-OST0000 complete [ 466.399919] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 466.404123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 466.410904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 469.380766] Lustre: server umount lustre-MDT0000 complete [ 470.644856] LustreError: 34495:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772305160 with bad export cookie 8629718739963038580 [ 470.647256] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 470.648352] LustreError: 34495:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 476.857190] Lustre: server umount lustre-MDT0001 complete [ 481.377322] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 482.364754] Key type lgssc unregistered [ 482.487416] LNet: 38343:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 482.491377] LNet: Removed LNI 192.168.204.147@tcp [ 482.793317] Key type .llcrypt unregistered [ 482.794916] Key type ._llcrypt unregistered [ 490.859798] alg: No test for adler32 (adler32-zlib) [ 491.611430] Key type ._llcrypt registered [ 491.612672] Key type .llcrypt registered [ 491.649321] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 492.096198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 492.274453] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 492.353461] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 492.355592] LNet: Accept secure, port 988 [ 493.935193] Key type lgssc registered [ 494.246922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 496.599749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 497.724535] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 497.736349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 498.948473] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 501.402414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 501.502328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 502.695409] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 504.950220] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 506.716389] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 508.681484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 508.786337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 510.074631] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 512.253880] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 512.867924] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 520.305322] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 521.409140] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 523.875355] LNetError: 120-3: Refusing connection from 192.168.204.147 for 127.0.0.2@tcp: No matching NI [ 523.877759] LNetError: 39107:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 523.879995] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 532.102777] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-02-2814h00m22s [ 532.589257] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 533.841422] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.47@tcp (stopping) [ 536.545380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 536.546370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 536.549790] Lustre: Skipped 2 previous similar messages [ 536.550850] LNetError: 120-3: Refusing connection from 192.168.204.147 for 127.0.0.2@tcp: No matching NI [ 536.552747] Lustre: Skipped 2 previous similar messages [ 536.557212] LNetError: 39104:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 536.559438] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 538.955862] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.47@tcp (stopping) [ 539.462895] Lustre: server umount lustre-MDT0000 complete [ 540.580822] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 541.664416] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.666836] LNetError: 120-3: Refusing connection from 192.168.204.147 for 127.0.0.2@tcp: No matching NI [ 541.668973] LustreError: Skipped 2 previous similar messages [ 541.673345] LNetError: 39105:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 541.676790] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 542.756368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 542.796599] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 542.798321] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 542.798856] LustreError: 11-0: MGC192.168.204.147@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 542.801321] LustreError: Skipped 2 previous similar messages [ 542.806395] Lustre: Evicted from MGS (at 192.168.204.147@tcp) after server handle changed from 0xb5adc68365a94123 to 0xb5adc68365a9456e [ 542.813028] Lustre: MGC192.168.204.147@tcp: Connection restored to 192.168.204.147@tcp (at 0@lo) [ 542.887946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 542.915934] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 544.303339] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 545.415505] Lustre: DEBUG MARKER: Wait for df (28805) ... [ 545.917461] Lustre: DEBUG MARKER: done [ 548.320393] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 548.322188] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 548.322583] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 548.323595] LNetError: 120-3: Refusing connection from 192.168.204.147 for 127.0.0.2@tcp: No matching NI [ 548.323685] LNetError: 39106:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 548.323721] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 548.323973] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 548.324879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 548.327570] Lustre: Skipped 1 previous similar message [ 548.346637] Lustre: Skipped 1 previous similar message [ 548.954459] Lustre: server umount lustre-OST0000 complete [ 553.440444] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 553.440660] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 553.443391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 553.447249] Lustre: Skipped 1 previous similar message [ 553.450382] Lustre: Skipped 1 previous similar message [ 556.358057] Lustre: server umount lustre-MDT0000 complete [ 557.602071] LustreError: 42501:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772305247 with bad export cookie 13091337959484507502 [ 557.605070] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 557.606676] LustreError: 42501:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 562.330153] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 563.317268] Key type lgssc unregistered [ 563.432402] LNet: 44892:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 563.436507] LNet: Removed LNI 192.168.204.147@tcp [ 563.712779] Key type .llcrypt unregistered [ 563.714073] Key type ._llcrypt unregistered [ 574.426800] alg: No test for adler32 (adler32-zlib) [ 575.178391] Key type ._llcrypt registered [ 575.179516] Key type .llcrypt registered [ 575.212661] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 580.504617] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 580.880965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 581.000057] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 581.064186] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 581.065490] LNet: Accept secure, port 988 [ 582.639092] Key type lgssc registered [ 582.942675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 585.524350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 587.481473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 589.240051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 591.020114] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 593.062020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 593.084695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 594.161454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 594.169617] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 594.202493] Lustre: lustre-MDT0000: new disk, initializing [ 594.223658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 594.229305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 595.494442] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 599.261171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 599.282231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 599.318644] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 599.336302] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 599.338368] Lustre: Skipped 1 previous similar message [ 599.369238] Lustre: lustre-MDT0001: new disk, initializing [ 599.395662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 599.406472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 599.409308] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 600.676451] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 604.188204] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 605.799462] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 607.843072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 607.870376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 607.873143] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 607.983560] Lustre: lustre-OST0000: new disk, initializing [ 607.985371] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 608.010671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 609.435599] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 610.357347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 610.360452] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 613.137151] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 614.608423] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 614.676455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 616.096870] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 616.177262] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 618.902557] Lustre: server umount lustre-OST0000 complete [ 620.512179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 620.512720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 620.516652] Lustre: Skipped 1 previous similar message [ 625.632123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 625.634556] Lustre: Skipped 2 previous similar messages [ 626.655135] Lustre: 52669:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772305310/real 1772305310] req@000000009161cb6d x1858396771856960/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1772305316 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 626.723382] Lustre: server umount lustre-MDT0000 complete [ 627.552084] Lustre: 47219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772305310/real 1772305310] req@0000000052774ed7 x1858396771857088/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1772305317 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 627.563749] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 627.570201] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 627.574266] LustreError: Skipped 1 previous similar message [ 627.981970] LustreError: 48986:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772305318 with bad export cookie 7843928228990042615 [ 627.985061] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 627.986208] LustreError: 48986:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 628.105944] Lustre: server umount lustre-MDT0001 complete [ 630.787329] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 14:02:01 (1772305321) [ 632.986373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 633.144107] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 633.167328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 634.455971] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 637.180287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 637.286307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 638.640262] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 641.083776] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 642.899364] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 645.132581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 646.613548] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 649.016676] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 655.944818] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-02-2814h02m26s [ 656.463085] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 658.609211] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 659.358432] Lustre: DEBUG MARKER: Creating a test file and stat it [ 659.371224] LustreError: 53535:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 sleeping for 40000ms [ 663.015127] LNetError: 120-3: Refusing connection from 192.168.204.147 for 127.0.0.2@tcp: No matching NI [ 663.017933] LNetError: 47210:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 663.020838] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 679.767214] Lustre: lustre-MDT0000: Client e1c4be9e-d9d4-49f2-aa89-567d32799db7 (at 192.168.204.47@tcp) reconnecting [ 699.415083] LustreError: 53535:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 awake [ 700.067216] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 700.806300] Lustre: DEBUG MARKER: done [ 703.968849] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 703.970985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 703.974363] Lustre: Skipped 1 previous similar message [ 703.979535] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 708.434600] Lustre: server umount lustre-OST0000 complete [ 714.208457] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 714.208589] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 714.209436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 714.209440] Lustre: Skipped 1 previous similar message [ 714.217575] Lustre: Skipped 1 previous similar message [ 714.219996] LNetError: 120-3: Refusing connection from 192.168.204.147 for 127.0.0.2@tcp: No matching NI [ 714.222697] LNetError: 47207:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 714.226495] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 715.716982] Lustre: server umount lustre-MDT0000 complete [ 716.901092] LustreError: 53521:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772305407 with bad export cookie 7843928228990043448 [ 716.903802] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 716.904349] LustreError: 53521:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 721.414385] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 722.353569] Key type lgssc unregistered [ 722.448342] LNet: 57952:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.451235] LNet: Removed LNI 192.168.204.147@tcp [ 722.732228] Key type .llcrypt unregistered [ 722.733617] Key type ._llcrypt unregistered [ 733.524332] alg: No test for adler32 (adler32-zlib) [ 734.275296] Key type ._llcrypt registered [ 734.276499] Key type .llcrypt registered [ 734.309307] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 739.922739] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 740.367534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 740.494170] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 740.565794] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 740.567251] LNet: Accept secure, port 988 [ 742.143187] Key type lgssc registered [ 742.452390] Lustre: Echo OBD driver; http://www.lustre.org/ [ 745.190067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 747.184606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 749.059475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 751.035751] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 753.165611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 753.186355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 754.274179] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 754.283207] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 754.322422] Lustre: lustre-MDT0000: new disk, initializing [ 754.346242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 754.352552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 755.700264] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 759.714139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 759.736289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 759.771413] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 759.787264] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 759.789135] Lustre: Skipped 1 previous similar message [ 759.823644] Lustre: lustre-MDT0001: new disk, initializing [ 759.853128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 759.864889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 759.867588] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 761.194070] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 764.886468] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 766.677781] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 768.837238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 768.862586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 768.866239] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 768.952040] Lustre: lustre-OST0000: new disk, initializing [ 768.954515] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 768.979023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 769.715734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 769.718838] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 770.302202] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 773.939740] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 775.346029] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 775.425367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 776.781292] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 776.855895] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 779.540768] Lustre: server umount lustre-OST0000 complete [ 785.376479] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 785.376661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 785.377228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 785.385312] Lustre: Skipped 1 previous similar message [ 785.887180] Lustre: 65716:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772305470/real 1772305470] req@00000000f03e747e x1858396938581248/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1772305476 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 787.423085] Lustre: 60265:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772305470/real 1772305470] req@0000000041511189 x1858396938580416/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1772305477 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 787.429526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 787.434487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 787.436598] Lustre: Skipped 2 previous similar messages [ 787.971285] Lustre: server umount lustre-MDT0000 complete [ 789.176446] LustreError: 62031:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772305479 with bad export cookie 15088478539841185292 [ 789.178857] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 789.181117] LustreError: 62031:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 789.303511] Lustre: server umount lustre-MDT0001 complete [ 791.931638] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 14:04:42 (1772305482) [ 792.423305] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 792.946064] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 14:04:43 (1772305483) [ 796.497746] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 797.959119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 798.091681] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 798.105554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 798.418404] Lustre: Failing over lustre-MDT0000 [ 798.421124] LustreError: 67049:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 798.425276] LustreError: 67049:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 798.428258] LustreError: 67049:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 804.831082] Lustre: 67100:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772305488/real 1772305488] req@00000000cbf3ff2f x1858396938584640/t0(0) o251->MGC192.168.204.147@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772305494 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 804.882304] Lustre: server umount lustre-MDT0000 complete [ 806.857227] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 14:04:57 (1772305497) [ 808.942900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 809.076878] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 809.090021] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 810.584076] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 813.239271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 814.749709] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 817.051351] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 818.799767] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 820.852448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 820.952424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 820.954921] Lustre: Skipped 1 previous similar message [ 822.306867] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 824.620360] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 831.289871] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 833.505302] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 833.506616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 833.509069] Lustre: Skipped 2 previous similar messages [ 833.511928] Lustre: Skipped 2 previous similar messages [ 838.624439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 838.626985] Lustre: Skipped 2 previous similar messages [ 838.977857] Lustre: server umount lustre-MDT0000 complete [ 840.161852] LustreError: 69618:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772305530 with bad export cookie 15088478539841186405 [ 840.164712] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 840.164752] LustreError: 69618:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 841.331780] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 842.832547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 845.509630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 846.816458] Lustre: Evicted from MGS (at 192.168.204.147@tcp) after server handle changed from 0xd1650b4a3056aa65 to 0xd1650b4a3056b1ea [ 846.820151] Lustre: MGC192.168.204.147@tcp: Connection restored to 192.168.204.147@tcp (at 0@lo) [ 846.922109] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 846.944987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 848.330560] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 848.993366] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 848.993943] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 850.015101] Lustre: 60268:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772305533/real 1772305533] req@00000000276513e0 x1858396938599040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772305540 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 851.009231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 851.139629] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:65 [ 852.428630] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 854.768423] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 856.547968] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 856.629260] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 859.616304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 859.617489] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 859.619335] Lustre: Skipped 1 previous similar message [ 859.623416] Lustre: Skipped 1 previous similar message [ 864.391944] Lustre: server umount lustre-MDT0000 complete [ 864.393464] Lustre: Skipped 1 previous similar message [ 864.736138] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 864.739879] LustreError: Skipped 3 previous similar messages [ 865.589101] LustreError: 67649:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772305555 with bad export cookie 15088478539841188330 [ 865.592255] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 865.593249] LustreError: 67649:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 867.820897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 870.713861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 870.880575] Lustre: Evicted from MGS (at 192.168.204.147@tcp) after server handle changed from 0xd1650b4a3056b1ea to 0xd1650b4a3056b7e7 [ 870.884324] Lustre: MGC192.168.204.147@tcp: Connection restored to (at 0@lo) [ 871.004862] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 871.023244] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 872.322228] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 874.016338] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 874.020152] LustreError: Skipped 4 previous similar messages [ 875.076909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 875.233985] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:97 [ 875.872095] Lustre: 60268:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772305559/real 1772305559] req@00000000c041d191 x1858396938609728/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772305566 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 875.878732] Lustre: 60268:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 876.538243] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 878.880968] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 880.613864] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 880.665540] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 885.729788] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 885.730309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 885.734024] Lustre: Skipped 1 previous similar message [ 888.455495] Lustre: server umount lustre-MDT0000 complete [ 888.457290] Lustre: Skipped 1 previous similar message [ 889.697049] LustreError: 67648:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772305579 with bad export cookie 15088478539841189863 [ 889.699649] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 889.701350] LustreError: 67648:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 891.950285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 893.048605] Lustre: DEBUG MARKER: files compared the same [ 899.039135] Lustre: 76787:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772305583/real 1772305583] req@000000009dac6e49 x1858396938618752/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772305589 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 899.046695] Lustre: 76787:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 905.391780] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 906.329846] Key type lgssc unregistered [ 906.442302] LNet: 77694:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.445218] LNet: Removed LNI 192.168.204.147@tcp [ 906.706201] Key type .llcrypt unregistered [ 906.707749] Key type ._llcrypt unregistered [ 915.014284] alg: No test for adler32 (adler32-zlib) [ 915.765343] Key type ._llcrypt registered [ 915.766269] Key type .llcrypt registered [ 915.800126] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 916.310181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 916.457137] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 916.532534] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 916.534094] LNet: Accept secure, port 988 [ 918.111086] Key type lgssc registered [ 918.513175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 921.211129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 922.378761] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 922.394874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 923.772226] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug +malloc all 8 [ 926.575073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 926.690111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 928.015650] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug +malloc all 8 [ 930.337151] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 932.105327] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 934.215539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 934.302665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 935.563991] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug +malloc all 8 [ 937.378963] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 937.979434] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 941.474964] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:129 [ 946.656141] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 946.657323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 946.657716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 946.664070] Lustre: Skipped 1 previous similar message [ 951.461405] Lustre: server umount lustre-OST0000 complete [ 956.895778] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 956.896049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 956.899010] Lustre: Skipped 1 previous similar message [ 956.901881] Lustre: Skipped 1 previous similar message [ 962.015530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 962.018189] Lustre: Skipped 1 previous similar message [ 967.135150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 967.135589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 967.141067] Lustre: Skipped 1 previous similar message [ 967.176718] Lustre: server umount lustre-MDT0000 complete [ 968.375486] LustreError: 81714:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772305658 with bad export cookie 7054117402063748841 [ 968.376818] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 968.380425] LustreError: 81714:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 968.491608] Lustre: server umount lustre-MDT0001 complete [ 972.891447] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 973.808301] Key type lgssc unregistered [ 973.913507] LNet: 82807:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 973.917473] LNet: Removed LNI 192.168.204.147@tcp [ 974.211186] Key type .llcrypt unregistered [ 974.212522] Key type ._llcrypt unregistered [ 982.452471] alg: No test for adler32 (adler32-zlib) [ 983.203380] Key type ._llcrypt registered [ 983.204631] Key type .llcrypt registered [ 983.241070] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 983.664380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 983.790957] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 983.860719] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 983.862821] LNet: Accept secure, port 988 [ 985.439090] Key type lgssc registered [ 985.742677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 988.172047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1005.663304] LustreError: 84242:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.147@tcp: failed processing log, type 1: rc = -5 [ 1036.383167] LustreError: 84242:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.147@tcp: failed processing log, type 4: rc = -110 [ 1066.079164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1067.376079] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1069.948835] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1072.686109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1072.724333] Lustre: *** cfs_fail_loc=706, val=0*** [ 1088.040081] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1088.066654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1088.084692] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 1089.402647] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1092.339996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1092.447646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1092.486580] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:161 [ 1093.848943] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1096.186778] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1097.976279] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1102.817577] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1102.818042] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1102.818957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1102.825282] Lustre: Skipped 1 previous similar message [ 1104.793105] Lustre: server umount lustre-OST0000 complete [ 1107.937226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1107.937698] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1107.937967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1107.937971] Lustre: Skipped 1 previous similar message [ 1107.943488] Lustre: Skipped 1 previous similar message [ 1112.135012] Lustre: server umount lustre-MDT0000 complete [ 1113.056075] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1113.061748] LustreError: Skipped 1 previous similar message [ 1113.376902] LustreError: 84248:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772305803 with bad export cookie 4274909417040252002 [ 1113.379042] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1113.380841] LustreError: 84248:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1113.493626] Lustre: server umount lustre-MDT0001 complete [ 1116.855168] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 1117.781496] Key type lgssc unregistered [ 1117.892380] LNet: 87980:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1117.895428] LNet: Removed LNI 192.168.204.147@tcp [ 1118.182656] Key type .llcrypt unregistered [ 1118.183602] Key type ._llcrypt unregistered [ 1126.735914] alg: No test for adler32 (adler32-zlib) [ 1127.487371] Key type ._llcrypt registered [ 1127.488905] Key type .llcrypt registered [ 1127.524292] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 1127.941643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1128.079544] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 1128.150716] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 1128.152223] LNet: Accept secure, port 988 [ 1129.727080] Key type lgssc registered [ 1130.041473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1132.525379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1134.903117] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1137.310886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1137.457547] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1137.482474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1138.844591] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1142.311970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1142.469694] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1142.475229] LustreError: Skipped 3 previous similar messages [ 1142.479174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1143.522785] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1143.527235] LustreError: Skipped 2 previous similar messages [ 1143.527831] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:193 [ 1143.928460] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1146.611334] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1146.617310] LustreError: Skipped 2 previous similar messages [ 1146.686778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1146.707927] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:193 [ 1148.125613] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1157.091701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1157.092977] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1157.093259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1157.095360] Lustre: Skipped 1 previous similar message [ 1161.958907] Lustre: server umount lustre-OST0000 complete [ 1167.328753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1167.329171] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1167.329540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1167.329545] Lustre: Skipped 1 previous similar message [ 1167.332854] Lustre: Skipped 1 previous similar message [ 1169.283468] Lustre: server umount lustre-MDT0000 complete [ 1170.635479] Lustre: server umount lustre-MDT0001 complete [ 1177.055161] Lustre: 92843:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772305861/real 1772305861] req@00000000402bcd66 x1858397345426048/t0(0) o251->MGC192.168.204.147@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772305867 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1177.066450] Lustre: server umount MGS complete [ 1180.537490] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 14:11:11 (1772305871) [ 1189.994624] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 1193.324551] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 1197.031095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1199.276422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1201.528758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1203.511974] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1205.570455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1205.594701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1206.963720] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1210.648260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1210.676284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1210.773242] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1210.785894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1210.820989] Lustre: lustre-MDT0001: new disk, initializing [ 1210.843432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1210.852140] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1210.854062] LustreError: 97535:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osd get update log failed: rc = -115 [ 1212.188613] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1216.252113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1216.292669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1216.299391] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1216.417619] Lustre: lustre-OST0000: new disk, initializing [ 1216.420595] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1216.447174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1218.032672] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1221.681650] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1223.870521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1223.885863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 1223.886318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1223.887683] Lustre: Skipped 1 previous similar message [ 1223.916945] LustreError: 98347:0:(fid_handler.c:334:__seq_server_alloc_meta()) srv-lustre-OST0000: Allocated super-sequence failed: rc = -11 [ 1223.934436] Lustre: lustre-MDT0000: new disk, initializing [ 1223.979669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1225.325776] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1225.951761] LustreError: 98347:0:(fid_handler.c:334:__seq_server_alloc_meta()) srv-lustre-OST0000: Allocated super-sequence failed: rc = -11 [ 1227.999674] LustreError: 98347:0:(fid_handler.c:334:__seq_server_alloc_meta()) srv-lustre-OST0000: Allocated super-sequence failed: rc = -11 [ 1229.286746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1229.294253] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1230.050291] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1249.760469] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1249.762281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1249.768610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1249.770443] Lustre: Skipped 3 previous similar messages [ 1254.881664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1254.884923] Lustre: Skipped 1 previous similar message [ 1255.261478] Lustre: server umount lustre-OST0000 complete [ 1260.000535] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1260.001241] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1260.009113] Lustre: Skipped 2 previous similar messages [ 1270.142629] Lustre: server umount lustre-MDT0001 complete [ 1270.143865] Lustre: Skipped 1 previous similar message [ 1276.895171] Lustre: 100576:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772305961/real 1772305961] req@00000000095954df x1858397345441472/t0(0) o251->MGC192.168.204.147@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772305967 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1280.429628] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 14:12:51 (1772305971) [ 1286.915694] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 1287.792639] Key type lgssc unregistered [ 1287.899389] LNet: 101982:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1287.902326] LNet: Removed LNI 192.168.204.147@tcp [ 1288.167502] Key type .llcrypt unregistered [ 1288.168567] Key type ._llcrypt unregistered [ 1294.390652] alg: No test for adler32 (adler32-zlib) [ 1295.141286] Key type ._llcrypt registered [ 1295.142753] Key type .llcrypt registered [ 1295.178330] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 1295.533745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1295.739294] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 1295.810149] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 1295.811752] LNet: Accept secure, port 988 [ 1297.391115] Key type lgssc registered [ 1297.710244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1300.243715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1301.376770] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1301.390783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1302.674650] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1305.261562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1305.370973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1306.686555] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1309.056617] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1311.014197] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1313.752677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1313.889590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1315.221094] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1315.940108] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 1317.541217] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1325.538875] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1325.541003] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1325.546826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1325.553676] Lustre: Skipped 1 previous similar message [ 1325.562144] Lustre: Skipped 1 previous similar message [ 1330.657384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1330.660534] Lustre: Skipped 1 previous similar message [ 1330.902703] Lustre: server umount lustre-OST0000 complete [ 1335.776506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1335.776981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1335.779819] Lustre: Skipped 1 previous similar message [ 1335.782609] Lustre: Skipped 1 previous similar message [ 1338.252214] Lustre: server umount lustre-MDT0000 complete [ 1339.558137] LustreError: 104032:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772306029 with bad export cookie 11366211555469168798 [ 1339.561078] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1339.562167] LustreError: 104032:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1339.704796] Lustre: server umount lustre-MDT0001 complete [ 1344.179548] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 1345.243269] Key type lgssc unregistered [ 1345.353316] LNet: 106898:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1345.355610] LNet: Removed LNI 192.168.204.147@tcp [ 1345.641950] Key type .llcrypt unregistered [ 1345.642956] Key type ._llcrypt unregistered [ 1351.481308] alg: No test for adler32 (adler32-zlib) [ 1352.233317] Key type ._llcrypt registered [ 1352.234242] Key type .llcrypt registered [ 1352.267634] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 1352.676102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1352.813568] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 1352.888815] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 1352.890254] LNet: Accept secure, port 988 [ 1354.471068] Key type lgssc registered [ 1354.795680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1356.756744] LustreError: 108025:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 1356.756857] LustreError: 108024:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 1356.762449] LustreError: 108025:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4996 [ 1356.764654] LustreError: 108025:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1356.769353] LustreError: 108025:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1356.769952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1357.909617] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1357.923311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1360.279176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1360.373734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1361.716140] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1363.323047] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing lsmod [ 1364.251025] LustreError: 109334:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 1364.251259] LustreError: 109335:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 1364.255880] LustreError: 109334:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4998 [ 1364.257690] LustreError: 109334:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1364.260691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1364.262074] LustreError: 109334:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1364.377477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1365.376719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1365.378505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1365.381171] Lustre: Skipped 2 previous similar messages [ 1365.385306] Lustre: Skipped 2 previous similar messages [ 1370.593469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1370.595738] Lustre: Skipped 2 previous similar messages [ 1371.404027] Lustre: server umount lustre-MDT0000 complete [ 1372.593948] LustreError: 109526:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772306062 with bad export cookie 12524165608795137945 [ 1372.596640] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1372.597041] LustreError: 109526:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1372.706515] Lustre: server umount lustre-MDT0001 complete [ 1380.319137] Lustre: 109930:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772306064/real 1772306064] req@0000000095955e1d x1858397581352896/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772306070 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1380.393903] Lustre: server umount lustre-OST0000 complete [ 1383.049407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1383.194713] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1383.214411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1384.560704] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1387.254960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1387.374223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1388.702324] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1391.014826] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1392.764073] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1394.862458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1396.267645] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1398.053759] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 1398.567677] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1407.457821] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1407.457901] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1407.460491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1407.466801] Lustre: Skipped 1 previous similar message [ 1407.469854] Lustre: Skipped 1 previous similar message [ 1411.864634] Lustre: server umount lustre-OST0000 complete [ 1413.088609] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1413.090567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1413.093957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1417.696054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1417.696412] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1419.202793] Lustre: server umount lustre-MDT0000 complete [ 1420.401922] LustreError: 110386:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772306110 with bad export cookie 12524165608795138617 [ 1420.404242] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1420.406355] LustreError: 110386:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1424.961364] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 1425.873704] Key type lgssc unregistered [ 1425.981261] LNet: 114058:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1425.984469] LNet: Removed LNI 192.168.204.147@tcp [ 1426.248018] Key type .llcrypt unregistered [ 1426.249488] Key type ._llcrypt unregistered [ 1434.541461] alg: No test for adler32 (adler32-zlib) [ 1435.293473] Key type ._llcrypt registered [ 1435.294479] Key type .llcrypt registered [ 1435.332874] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 1435.741113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1435.863395] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 1435.937565] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 1435.938963] LNet: Accept secure, port 988 [ 1437.511136] Key type lgssc registered [ 1437.890312] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1440.593577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1441.731195] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1441.747571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1443.101762] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1445.866365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1445.978289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1447.326300] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1449.980343] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1451.761109] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1453.942589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1454.042809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1455.343684] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1457.124905] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 1457.696997] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1464.561706] Lustre: Setting parameter lustre-client.llite.some_wrong_param in log lustre-client [ 1465.367487] Lustre: Modifying parameter lustre-client.llite.some_wrong_param in log lustre-client [ 1466.337092] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1466.338489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1466.339939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1466.348929] Lustre: Skipped 1 previous similar message [ 1471.457196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1471.459149] Lustre: Skipped 2 previous similar messages [ 1472.402558] Lustre: server umount lustre-OST0000 complete [ 1476.576637] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1476.576795] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1476.577183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1476.590062] Lustre: Skipped 1 previous similar message [ 1479.749227] Lustre: server umount lustre-MDT0000 complete [ 1480.932038] LustreError: 115499:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772306171 with bad export cookie 4603165612604648283 [ 1480.934159] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1480.935647] LustreError: 115499:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1481.057581] Lustre: server umount lustre-MDT0001 complete [ 1486.150342] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 1487.197727] Key type lgssc unregistered [ 1487.318405] LNet: 119276:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1487.322555] LNet: Removed LNI 192.168.204.147@tcp [ 1487.654849] Key type .llcrypt unregistered [ 1487.656272] Key type ._llcrypt unregistered [ 1494.381403] alg: No test for adler32 (adler32-zlib) [ 1495.132400] Key type ._llcrypt registered [ 1495.133601] Key type .llcrypt registered [ 1495.174641] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 1495.538750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1495.829254] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 1495.912748] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 1495.914534] LNet: Accept secure, port 988 [ 1497.495116] Key type lgssc registered [ 1497.811336] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1500.297871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1501.440616] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1501.455952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1502.797063] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1505.569841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1505.695088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1507.068180] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1509.467946] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1511.407429] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1513.559605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1513.665834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1515.016082] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1516.777930] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 1517.484434] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1526.241874] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1526.242135] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1526.242717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1526.246539] Lustre: Skipped 1 previous similar message [ 1531.220129] Lustre: server umount lustre-OST0000 complete [ 1536.480735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1536.480782] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1536.481311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1536.481315] Lustre: Skipped 1 previous similar message [ 1536.486765] Lustre: Skipped 1 previous similar message [ 1538.561914] Lustre: server umount lustre-MDT0000 complete [ 1539.854068] LustreError: 120521:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772306229 with bad export cookie 8113311674999385633 [ 1539.856492] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1539.857575] LustreError: 120521:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1539.980520] Lustre: server umount lustre-MDT0001 complete [ 1544.466144] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 1545.428253] Key type lgssc unregistered [ 1545.536316] LNet: 124191:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1545.538611] LNet: Removed LNI 192.168.204.147@tcp [ 1545.829347] Key type .llcrypt unregistered [ 1545.830699] Key type ._llcrypt unregistered [ 1554.401663] alg: No test for adler32 (adler32-zlib) [ 1555.152361] Key type ._llcrypt registered [ 1555.154019] Key type .llcrypt registered [ 1555.189915] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 1555.603369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1555.739266] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 1555.822271] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 1555.823863] LNet: Accept secure, port 988 [ 1557.407099] Key type lgssc registered [ 1557.746340] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1560.258030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1561.395379] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1561.410484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1562.809682] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1565.537933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1565.648913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1567.004048] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1569.381991] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1571.200777] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1573.327317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1573.432854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1574.871454] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1576.486816] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 1577.446318] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1584.828217] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 1585.533527] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 1586.066889] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 1586.069809] Lustre: Skipped 2 previous similar messages [ 1587.399593] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 1587.401951] Lustre: Skipped 2 previous similar messages [ 1594.868724] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1594.879266] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1594.881644] Lustre: Skipped 1 previous similar message [ 1597.357645] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 1597.359557] Lustre: Skipped 2 previous similar messages [ 1602.038854] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.47@tcp 192.168.0.[2,10]@tcp [ 1606.113168] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1606.113959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1606.117234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1606.119625] Lustre: Skipped 1 previous similar message [ 1609.881209] Lustre: server umount lustre-OST0000 complete [ 1611.232210] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1611.232593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1611.237203] Lustre: Skipped 1 previous similar message [ 1611.239138] Lustre: Skipped 1 previous similar message [ 1616.351889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1616.354095] Lustre: Skipped 2 previous similar messages [ 1617.219079] Lustre: server umount lustre-MDT0000 complete [ 1618.431218] LustreError: 128554:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772306308 with bad export cookie 7207539415237765003 [ 1618.434159] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1618.436751] LustreError: 128554:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1618.557963] Lustre: server umount lustre-MDT0001 complete [ 1622.981690] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 1623.891192] Key type lgssc unregistered [ 1624.003352] LNet: 130601:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1624.005721] LNet: Removed LNI 192.168.204.147@tcp [ 1624.289458] Key type .llcrypt unregistered [ 1624.290599] Key type ._llcrypt unregistered [ 1633.993968] alg: No test for adler32 (adler32-zlib) [ 1634.745274] Key type ._llcrypt registered [ 1634.746433] Key type .llcrypt registered [ 1634.779976] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 1635.183418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1635.340691] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 1635.417839] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 1635.419383] LNet: Accept secure, port 988 [ 1636.999057] Key type lgssc registered [ 1637.330491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1639.782368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1640.920718] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1640.932101] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1640.936567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1640.940654] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.204.47@tcp 192.168.0.[2,10]@tcp [ 1642.323470] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1645.093631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1645.208129] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1645.217714] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.47@tcp 192.168.0.[2,10]@tcp [ 1645.230492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1646.632122] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1649.248133] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1651.206739] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1653.896918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1654.021681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1655.477694] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1656.105535] Lustre: lustre-OST0000: deleting orphan objects from 0x0:167 to 0x0:193 [ 1658.007064] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1666.527922] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1666.530937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1666.537121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1670.624634] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1670.625594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1670.626992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1672.023248] Lustre: server umount lustre-OST0000 complete [ 1675.744925] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1675.745712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1675.749586] Lustre: Skipped 1 previous similar message [ 1675.751950] Lustre: Skipped 1 previous similar message [ 1679.453400] Lustre: server umount lustre-MDT0000 complete [ 1680.864943] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1680.874271] LustreError: Skipped 1 previous similar message [ 1681.057301] LustreError: 132189:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772306371 with bad export cookie 9785657199307652348 [ 1681.060386] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1681.061758] LustreError: 132189:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1681.199635] Lustre: server umount lustre-MDT0001 complete [ 1685.641297] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 1686.840549] Key type lgssc unregistered [ 1686.958315] LNet: 135914:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1686.961410] LNet: Removed LNI 192.168.204.147@tcp [ 1687.223142] Key type .llcrypt unregistered [ 1687.224337] Key type ._llcrypt unregistered [ 1696.773924] alg: No test for adler32 (adler32-zlib) [ 1697.526299] Key type ._llcrypt registered [ 1697.527604] Key type .llcrypt registered [ 1697.574196] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 1702.758849] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 1703.108826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1703.299559] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 1703.373791] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 1703.375186] LNet: Accept secure, port 988 [ 1704.959126] Key type lgssc registered [ 1705.280852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1708.140894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1710.944046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1712.850045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1714.786468] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1717.159302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1717.189863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1718.271081] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1718.281158] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1718.316668] Lustre: lustre-MDT0000: new disk, initializing [ 1718.340861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1718.349845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1719.723347] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1724.167962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1724.195134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1724.231262] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1724.252874] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1724.257455] Lustre: Skipped 1 previous similar message [ 1724.305233] Lustre: lustre-MDT0001: new disk, initializing [ 1724.339345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1724.353768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1724.357319] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1725.977911] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1730.209537] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1732.283169] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1734.820371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1734.847908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1734.851632] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1734.953441] Lustre: lustre-OST0000: new disk, initializing [ 1734.955682] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1734.988668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1736.461681] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1738.741078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1738.744134] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1740.163722] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1741.564109] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1741.633430] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1743.019442] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1743.095628] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1743.841623] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1743.841692] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1743.844194] LustreError: Skipped 1 previous similar message [ 1743.844880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1743.848259] Lustre: Skipped 1 previous similar message [ 1743.852941] Lustre: Skipped 1 previous similar message [ 1745.815912] Lustre: server umount lustre-OST0000 complete [ 1748.960847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1748.961685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1748.969188] Lustre: Skipped 1 previous similar message [ 1748.974146] Lustre: Skipped 1 previous similar message [ 1753.428877] Lustre: server umount lustre-MDT0000 complete [ 1754.080876] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1754.084342] LustreError: Skipped 1 previous similar message [ 1754.662950] LustreError: 139426:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772306444 with bad export cookie 7817716154607857807 [ 1754.665636] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1754.666991] LustreError: 139426:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1754.782493] Lustre: server umount lustre-MDT0001 complete [ 1757.572907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1757.727275] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1757.747573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1759.152219] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1761.929222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1762.036800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1763.367510] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1765.662240] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1767.402580] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1769.467504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1770.903345] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1773.228060] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1774.642617] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1774.717352] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1776.057933] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1778.167092] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 2 sec [ 1781.380020] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1781.406289] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1781.409207] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 1781.459641] Lustre: lustre-OST0001: new disk, initializing [ 1781.462039] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 1781.489691] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1781.491895] Lustre: Skipped 1 previous similar message [ 1782.332479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1782.337322] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1782.795712] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1786.004445] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 1786.077182] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1787.411244] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 1787.480764] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1789.204872] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff970d903e0800.ost_server_uuid 40 [ 1789.669632] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff970d903e0800.ost_server_uuid in FULL state after 0 sec [ 1792.481315] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1792.482077] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1792.484542] LustreError: Skipped 1 previous similar message [ 1792.488586] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1792.491909] Lustre: Skipped 1 previous similar message [ 1797.482822] Lustre: server umount lustre-OST0001 complete [ 1797.603192] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1797.607385] LustreError: Skipped 1 previous similar message [ 1799.647799] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1799.650315] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1799.653763] Lustre: Skipped 1 previous similar message [ 1799.655498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1802.722465] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1802.726636] LustreError: Skipped 1 previous similar message [ 1804.757732] Lustre: server umount lustre-OST0000 complete [ 1807.840336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1807.844640] Lustre: Skipped 1 previous similar message [ 1807.847026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1807.848594] Lustre: Skipped 3 previous similar messages [ 1812.959888] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1812.964461] LustreError: Skipped 1 previous similar message [ 1813.296993] LustreError: 147336:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772306503 with bad export cookie 7817716154607858661 [ 1813.299805] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1813.300217] LustreError: 147336:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1813.411473] Lustre: server umount lustre-MDT0001 complete [ 1813.412729] Lustre: Skipped 1 previous similar message [ 1820.557485] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 1821.476242] Key type lgssc unregistered [ 1821.591367] LNet: 150446:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1821.594277] LNet: Removed LNI 192.168.204.147@tcp [ 1821.871401] Key type .llcrypt unregistered [ 1821.872480] Key type ._llcrypt unregistered [ 1832.337392] alg: No test for adler32 (adler32-zlib) [ 1833.089373] Key type ._llcrypt registered [ 1833.090391] Key type .llcrypt registered [ 1833.124422] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 1838.745479] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 1839.155500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1839.284815] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 1839.359407] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 1839.361134] LNet: Accept secure, port 988 [ 1840.935122] Key type lgssc registered [ 1841.257664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1843.975696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1845.977221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1847.863083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1849.813898] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1851.936796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1851.963964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1853.045277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1853.054867] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1853.089330] Lustre: lustre-MDT0000: new disk, initializing [ 1853.113894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1853.119893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1854.414371] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1858.351684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1858.376557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1858.407409] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1858.424716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1858.427498] Lustre: Skipped 1 previous similar message [ 1858.467569] Lustre: lustre-MDT0001: new disk, initializing [ 1858.497541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1858.510608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1858.514462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1859.911698] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1863.564995] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1865.407367] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1867.630034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1867.653292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1867.657140] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1867.742970] Lustre: lustre-OST0000: new disk, initializing [ 1867.745371] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1867.773164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1868.405446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1868.408645] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1869.093421] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1872.766069] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1874.159897] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1874.238404] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1875.560771] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1875.633591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1878.354330] Lustre: server umount lustre-OST0000 complete [ 1884.128461] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1884.128507] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1884.128982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1884.139649] Lustre: Skipped 1 previous similar message [ 1885.663098] Lustre: 158207:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772306569/real 1772306569] req@00000000ca4e635d x1858398090966208/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1772306575 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1885.729772] Lustre: server umount lustre-MDT0000 complete [ 1886.175087] Lustre: 152762:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772306569/real 1772306569] req@00000000288373c4 x1858398090965376/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1772306576 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 1886.181462] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1886.185355] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1886.188734] LustreError: Skipped 1 previous similar message [ 1886.945970] LustreError: 154527:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772306577 with bad export cookie 5117190503506322024 [ 1886.948656] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1886.951587] LustreError: 154527:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1887.067566] Lustre: server umount lustre-MDT0001 complete [ 1889.782593] Lustre: DEBUG MARKER: == conf-sanity test 47: server restart does not make client loss lru_resize settings ========================================================== 14:23:00 (1772306580) [ 1894.785507] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 1897.517296] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 1900.554520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1902.584877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1904.503197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1906.491538] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1908.728195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1908.753026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1908.848637] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1908.860269] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1908.900909] Lustre: lustre-MDT0000: new disk, initializing [ 1908.928475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1908.935233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1910.291824] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1914.291982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1914.314314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1914.351952] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1914.367270] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1914.368986] Lustre: Skipped 1 previous similar message [ 1914.405130] Lustre: lustre-MDT0001: new disk, initializing [ 1914.434105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1914.446212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1914.449801] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1915.795306] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1919.457321] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1921.248471] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1923.379172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1923.403555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1923.405991] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1923.487593] Lustre: lustre-OST0000: new disk, initializing [ 1923.489795] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1923.513634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1924.840339] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1928.179452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1928.182838] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1928.518263] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1930.446678] Lustre: Failing over lustre-OST0000 [ 1930.508760] Lustre: server umount lustre-OST0000 complete [ 1940.431149] Lustre: 152759:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772306623/real 1772306623] req@00000000d2e8e529 x1858398090979264/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1772306630 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 1940.437884] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1943.601390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1943.724557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1943.732972] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1945.088586] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1945.760128] Lustre: 152762:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772306628/real 1772306628] req@000000007a257d29 x1858398090980224/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1772306635 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1945.766857] Lustre: 152762:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1946.307252] Lustre: Failing over lustre-MDT0000 [ 1946.336456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1946.337149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1946.341796] Lustre: Skipped 2 previous similar messages [ 1946.343376] Lustre: Skipped 1 previous similar message [ 1946.451268] Lustre: server umount lustre-MDT0000 complete [ 1947.360675] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1947.361578] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 1947.367290] LustreError: Skipped 2 previous similar messages [ 1948.492651] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1948.496971] LustreError: Skipped 1 previous similar message [ 1952.736517] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1952.741832] LustreError: Skipped 2 previous similar messages [ 1954.399116] Lustre: 152760:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772306637/real 1772306637] req@00000000b1f58b5e x1858398090983936/t0(0) o400->MGC192.168.204.147@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772306644 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1954.406863] Lustre: 152760:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1954.409333] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1959.443808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1959.756419] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1959.760508] LustreError: Skipped 5 previous similar messages [ 1959.906275] Lustre: Evicted from MGS (at 192.168.204.147@tcp) after server handle changed from 0x4703e9a6a17f25b0 to 0x4703e9a6a17f2a56 [ 1959.909825] Lustre: MGC192.168.204.147@tcp: Connection restored to (at 0@lo) [ 1959.971863] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 1960.012243] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1960.780564] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1961.316576] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 1965.034153] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1965.039940] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1965.043039] Lustre: Skipped 1 previous similar message [ 1965.059963] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 1970.145918] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1970.146159] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1970.149475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1970.151545] LustreError: Skipped 1 previous similar message [ 1970.153250] Lustre: Skipped 3 previous similar messages [ 1972.114551] Lustre: server umount lustre-OST0000 complete [ 1975.264586] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1975.265364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1975.271191] Lustre: Skipped 1 previous similar message [ 1980.384915] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1980.389750] LustreError: Skipped 4 previous similar messages [ 1980.667131] LustreError: 163593:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772306670 with bad export cookie 5117190503506324054 [ 1980.669384] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1980.670649] LustreError: 163593:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1985.269581] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 1986.217284] Key type lgssc unregistered [ 1986.335309] LNet: 167346:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1986.339830] LNet: Removed LNI 192.168.204.147@tcp [ 1986.598347] Key type .llcrypt unregistered [ 1986.599478] Key type ._llcrypt unregistered [ 1995.157541] alg: No test for adler32 (adler32-zlib) [ 1995.909321] Key type ._llcrypt registered [ 1995.910332] Key type .llcrypt registered [ 1995.946498] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 1996.290401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1996.494030] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 1996.570157] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 1996.571656] LNet: Accept secure, port 988 [ 1998.143116] Key type lgssc registered [ 1998.482134] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2000.965483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2002.097857] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2002.112052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2003.484749] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2006.248327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2006.359709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2007.726296] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2010.182112] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2011.938310] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2014.282470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2014.402552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2015.768039] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2017.511615] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 2018.461691] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2114.896441] Lustre: Failing over lustre-MDT0000 [ 2115.085206] Lustre: server umount lustre-MDT0000 complete [ 2116.948955] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2119.136929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2119.138166] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2119.142031] Lustre: Skipped 2 previous similar messages [ 2119.145413] LustreError: Skipped 1 previous similar message [ 2119.149981] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2122.061638] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2122.074918] LustreError: Skipped 1 previous similar message [ 2126.303161] Lustre: 168118:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772306809/real 1772306809] req@00000000003643c3 x1858398255891008/t0(0) o400->MGC192.168.204.147@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772306816 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2126.316246] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2126.324239] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2126.331293] LustreError: Skipped 6 previous similar messages [ 2129.212109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2132.453186] Lustre: Evicted from MGS (at 192.168.204.147@tcp) after server handle changed from 0xfebf32cc2888ddd7 to 0xfebf32cc288b49f1 [ 2132.466520] Lustre: MGC192.168.204.147@tcp: Connection restored to 192.168.204.147@tcp (at 0@lo) [ 2132.573893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2132.602791] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2133.983216] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2135.372272] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2138.084602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2138.091431] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2138.110160] Lustre: lustre-OST0000: deleting orphan objects from 0x0:68 to 0x0:97 [ 2139.892537] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2140.545428] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2153.441522] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2153.441883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2153.443175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2153.448737] Lustre: Skipped 1 previous similar message [ 2158.560772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2158.562708] Lustre: Skipped 4 previous similar messages [ 2159.432176] Lustre: server umount lustre-MDT0000 complete [ 2160.642261] LustreError: 168786:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772306850 with bad export cookie 18356446458620365297 [ 2160.645606] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2160.645891] LustreError: 168786:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2163.679832] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2163.679933] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2163.684960] LustreError: Skipped 8 previous similar messages [ 2163.690046] Lustre: Skipped 1 previous similar message [ 2163.691605] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2165.727991] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2166.841246] Lustre: server umount lustre-MDT0001 complete [ 2174.507747] Lustre: server umount lustre-OST0000 complete [ 2178.099497] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 2180.918359] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 2184.408336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2186.504125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2188.408359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2190.366515] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2192.569773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2192.591066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2192.670325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2192.681204] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2192.721192] Lustre: lustre-MDT0000: new disk, initializing [ 2192.744081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2192.749909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2194.165862] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2198.206136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2198.224619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2198.253542] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2198.268144] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2198.270010] Lustre: Skipped 1 previous similar message [ 2198.303832] Lustre: lustre-MDT0001: new disk, initializing [ 2198.339748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2198.343656] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2199.662399] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2203.474563] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2205.349385] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2207.647566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2207.675602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2207.678575] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2207.777147] Lustre: lustre-OST0000: new disk, initializing [ 2207.779484] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2207.809354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2207.812536] Lustre: Skipped 1 previous similar message [ 2209.166711] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2210.419791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2210.423070] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2212.854651] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2214.267849] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2214.348619] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2215.925846] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2216.030126] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2221.027670] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2221.027924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2221.033576] LustreError: Skipped 1 previous similar message [ 2221.043629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2221.051025] Lustre: Skipped 2 previous similar messages [ 2231.263107] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2231.266297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2231.268373] Lustre: Skipped 3 previous similar messages [ 2231.325941] Lustre: server umount lustre-OST0000 complete [ 2234.335734] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2234.337979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2234.341020] Lustre: Skipped 1 previous similar message [ 2239.876950] LustreError: 175658:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772306930 with bad export cookie 18356446458620366018 [ 2239.879805] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2239.880092] LustreError: 175658:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2239.995991] Lustre: server umount lustre-MDT0001 complete [ 2239.997559] Lustre: Skipped 1 previous similar message [ 2242.603684] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:28:53 (1772306933) [ 2247.607643] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 2250.147139] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 2253.336761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2255.284564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2257.149769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2259.116048] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2261.167731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2261.191330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2261.273144] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2261.282311] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2261.324846] Lustre: lustre-MDT0000: new disk, initializing [ 2261.346913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2261.353601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2262.660774] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2266.758292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2266.782405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2266.819336] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2266.835350] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2266.837179] Lustre: Skipped 1 previous similar message [ 2266.871494] Lustre: lustre-MDT0001: new disk, initializing [ 2266.908366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2266.912969] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2268.263057] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2272.112482] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2274.022470] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2276.219808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2276.243660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2276.246612] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2276.336530] Lustre: lustre-OST0000: new disk, initializing [ 2276.338496] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2277.716871] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2278.592288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2278.595641] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2281.927177] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2284.000457] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2284.000612] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2284.002581] LustreError: Skipped 1 previous similar message [ 2284.003172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2284.005883] Lustre: Skipped 2 previous similar messages [ 2284.010855] Lustre: Skipped 4 previous similar messages [ 2290.073172] Lustre: server umount lustre-OST0000 complete [ 2292.703737] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2298.478825] LustreError: 182637:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772306988 with bad export cookie 18356446458620366858 [ 2298.481135] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2298.482377] LustreError: 182637:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2301.150402] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:29:51 (1772306991) [ 2305.941281] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 2308.420662] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 2311.242139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2313.182785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2314.945046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2316.795582] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2318.891358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2318.911374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2318.991873] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2319.002102] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2319.037810] Lustre: lustre-MDT0000: new disk, initializing [ 2319.057955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2319.059665] Lustre: Skipped 2 previous similar messages [ 2319.064150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2320.379754] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2324.255751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2324.280243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2324.391607] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2325.649801] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2329.213888] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2330.967812] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2333.131921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2333.157793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2333.160165] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2334.557897] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2338.201987] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2338.802327] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2349.024834] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2349.025812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2349.027158] LustreError: Skipped 1 previous similar message [ 2349.031386] Lustre: Skipped 2 previous similar messages [ 2349.032055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2349.036069] Lustre: Skipped 6 previous similar messages [ 2351.523488] Lustre: server umount lustre-OST0000 complete [ 2351.524947] Lustre: Skipped 2 previous similar messages [ 2359.264319] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2359.267922] LustreError: Skipped 3 previous similar messages [ 2360.077996] LustreError: 189091:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772307050 with bad export cookie 18356446458620367922 [ 2360.080921] LustreError: 189091:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2360.081372] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2364.517390] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 2365.446720] Key type lgssc unregistered [ 2365.547351] LNet: 193058:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2365.549654] LNet: Removed LNI 192.168.204.147@tcp [ 2365.806661] Key type .llcrypt unregistered [ 2365.807680] Key type ._llcrypt unregistered [ 2374.201236] alg: No test for adler32 (adler32-zlib) [ 2374.953301] Key type ._llcrypt registered [ 2374.954248] Key type .llcrypt registered [ 2374.985390] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 2375.264765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2375.477634] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 2375.543148] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 2375.544512] LNet: Accept secure, port 988 [ 2377.119065] Key type lgssc registered [ 2377.396598] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2379.760651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2380.888860] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2380.903249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2382.348314] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2385.052498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2385.180845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2386.628225] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2389.010904] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2390.844147] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2392.950815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2393.052066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2394.345807] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2396.132476] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2396.744337] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2415.585151] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.585452] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2415.588799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2415.591850] Lustre: Skipped 1 previous similar message [ 2419.344381] Lustre: server umount lustre-OST0000 complete [ 2420.704105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2420.704564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2420.707416] Lustre: Skipped 2 previous similar messages [ 2420.710061] Lustre: Skipped 1 previous similar message [ 2425.823863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2425.825842] Lustre: Skipped 1 previous similar message [ 2426.627077] Lustre: server umount lustre-MDT0000 complete [ 2427.784062] LustreError: 196465:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772307117 with bad export cookie 6970152431616030021 [ 2427.786615] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2427.787875] LustreError: 196465:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2427.900974] Lustre: server umount lustre-MDT0001 complete [ 2432.102330] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 2432.980493] Key type lgssc unregistered [ 2433.096330] LNet: 198188:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2433.100210] LNet: Removed LNI 192.168.204.147@tcp [ 2433.349476] Key type .llcrypt unregistered [ 2433.350785] Key type ._llcrypt unregistered [ 2441.477954] alg: No test for adler32 (adler32-zlib) [ 2442.229293] Key type ._llcrypt registered [ 2442.230258] Key type .llcrypt registered [ 2442.264054] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 2442.576519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2442.779917] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 2442.859271] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 2442.860821] LNet: Accept secure, port 988 [ 2444.431080] Key type lgssc registered [ 2444.767727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2447.106741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2448.236407] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2448.249847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2449.622668] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2452.242403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2452.370292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2453.778214] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2456.053387] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2457.767625] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2459.912673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2460.019832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2461.307133] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2463.076841] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 2463.561025] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2472.416579] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2472.417791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2472.420160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2472.426238] Lustre: Skipped 1 previous similar message [ 2474.839591] Lustre: lustre-OST0000: Not available for connect from 192.168.204.47@tcp (stopping) [ 2474.842696] Lustre: Skipped 1 previous similar message [ 2476.504324] Lustre: server umount lustre-OST0000 complete [ 2478.975740] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff970d98902800.ost_server_uuid 40 [ 2479.479362] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff970d98902800.ost_server_uuid in DISCONN state after 0 sec [ 2479.990833] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2493.407671] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2493.411030] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2493.416455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2496.003637] Lustre: server umount lustre-MDT0000 complete [ 2497.139848] LustreError: 201598:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772307187 with bad export cookie 3038846769484757650 [ 2497.142279] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2497.142783] LustreError: 201598:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2497.261990] Lustre: server umount lustre-MDT0001 complete [ 2499.804106] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 14:33:10 (1772307190) [ 2501.875625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2502.013183] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2502.033846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2503.303779] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2505.817180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2507.205824] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2509.487988] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2511.173267] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2513.182622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2513.280362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2513.282268] Lustre: Skipped 1 previous similar message [ 2514.588918] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2516.734506] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2518.738282] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2518.761358] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2518.763549] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2518.810649] Lustre: lustre-OST0001: new disk, initializing [ 2518.812217] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2520.027552] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2521.896352] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 2522.875146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2522.877830] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2523.682979] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2527.199917] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2527.200022] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2527.202039] LustreError: Skipped 1 previous similar message [ 2527.206307] Lustre: Skipped 1 previous similar message [ 2527.207765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2527.211418] Lustre: Skipped 2 previous similar messages [ 2530.602340] Lustre: server umount lustre-OST0000 complete [ 2532.525291] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2532.596023] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2533.346986] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2533.353249] LustreError: Skipped 1 previous similar message [ 2533.914257] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2533.985842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2538.466235] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2538.469896] LustreError: Skipped 2 previous similar messages [ 2543.587330] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2543.590892] LustreError: Skipped 2 previous similar messages [ 2548.706499] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2548.706533] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2548.708660] LustreError: Skipped 1 previous similar message [ 2548.709073] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2548.711744] Lustre: Skipped 1 previous similar message [ 2548.712880] Lustre: Skipped 3 previous similar messages [ 2550.035606] Lustre: server umount lustre-OST0001 complete [ 2553.825258] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2553.829820] Lustre: Skipped 1 previous similar message [ 2558.945283] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.953072] LustreError: Skipped 3 previous similar messages [ 2559.128732] LustreError: 208081:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772307249 with bad export cookie 3038846769484758700 [ 2559.135928] LustreError: 208081:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2559.135970] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2559.326308] Lustre: server umount lustre-MDT0001 complete [ 2559.327771] Lustre: Skipped 1 previous similar message [ 2562.114364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2564.008940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2565.832039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2567.734693] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2571.004447] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 14:34:21 (1772307261) [ 2573.618742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2573.765275] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2573.787595] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2573.902337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2573.909300] Lustre: Skipped 1 previous similar message [ 2575.757228] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2579.732813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2579.777567] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2579.794165] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2581.952813] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2584.963892] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2586.701686] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2588.900894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2588.986536] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2590.388840] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2591.885886] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:129 [ 2592.773689] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2594.900882] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2594.922173] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2594.983161] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2594.984874] Lustre: Skipped 2 previous similar messages [ 2596.391834] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2599.090444] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2600.419118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2600.422988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2600.424942] Lustre: Skipped 5 previous similar messages [ 2601.991131] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3 to 0x0:33 [ 2614.239117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2614.261190] Lustre: server umount lustre-OST0000 complete [ 2614.607449] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.204.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.554958] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2627.556169] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2627.557165] LustreError: Skipped 2 previous similar messages [ 2627.560365] Lustre: Skipped 1 previous similar message [ 2627.562394] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2627.565395] Lustre: Skipped 9 previous similar messages [ 2639.612179] LustreError: 210051:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772307329 with bad export cookie 3038846769484760065 [ 2639.615538] LustreError: 210051:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2639.616199] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2641.715847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2642.873040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2643.995073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2645.167024] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2647.095176] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 14:35:37 (1772307337) [ 2651.880150] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 2654.470661] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 2657.980497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2660.355325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2662.279600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2664.191756] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2666.256081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2666.280810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2666.362952] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2666.371639] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2666.409596] Lustre: lustre-MDT0000: new disk, initializing [ 2666.431535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2666.438370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2667.770279] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2671.755491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2671.779784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2671.811379] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2671.827169] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2671.829960] Lustre: Skipped 1 previous similar message [ 2671.870623] Lustre: lustre-MDT0001: new disk, initializing [ 2671.907689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2671.910538] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2673.192138] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2676.644323] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2678.343810] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2680.487731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2680.510372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2680.512920] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2680.602319] Lustre: lustre-OST0000: new disk, initializing [ 2680.604609] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2682.167211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2682.171137] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2682.228647] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2685.936209] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2687.482568] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2687.562124] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2688.865196] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2688.930795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2692.577900] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2692.578320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2692.579038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2692.579042] Lustre: Skipped 4 previous similar messages [ 2692.581272] LustreError: Skipped 1 previous similar message [ 2692.588046] Lustre: Skipped 3 previous similar messages [ 2695.635299] Lustre: server umount lustre-OST0000 complete [ 2695.636510] Lustre: Skipped 3 previous similar messages [ 2704.219166] LustreError: 219113:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772307394 with bad export cookie 3038846769484761759 [ 2704.222992] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2704.223776] LustreError: 219113:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2707.096200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2707.252843] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2707.257085] LustreError: Skipped 7 previous similar messages [ 2708.613748] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2711.434973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2712.998253] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2715.321516] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2717.042472] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2719.100691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2720.512694] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2722.730610] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2724.066059] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2724.139650] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2725.499926] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2726.584172] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec [ 2727.391684] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2727.393749] LustreError: Skipped 1 previous similar message [ 2731.061505] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2731.127144] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2732.390908] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2738.655113] Lustre: 198961:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772307421/real 1772307421] req@0000000084948f14 x1858398723326464/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1772307428 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 2743.247081] Lustre: 198961:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772307426/real 1772307426] req@00000000540af755 x1858398723327296/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1772307433 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2762.842637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2762.942525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2762.945677] Lustre: Skipped 5 previous similar messages [ 2764.191319] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2766.490723] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2770.101564] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 2771.168538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2771.170248] Lustre: Skipped 6 previous similar messages [ 2777.102513] Lustre: server umount lustre-OST0000 complete [ 2777.104243] Lustre: Skipped 3 previous similar messages [ 2781.664393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2781.667696] Lustre: Skipped 4 previous similar messages [ 2785.574477] LustreError: 227057:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772307475 with bad export cookie 3038846769484762599 [ 2785.576824] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2785.579907] LustreError: 227057:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2788.204059] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 14:37:58 (1772307478) [ 2790.177090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2790.318494] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2791.631234] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2794.197481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2795.638972] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2797.855634] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2799.538398] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2801.662222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2803.091945] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2805.391744] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2806.713863] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2806.784617] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2808.047250] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2809.135410] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec [ 2811.153814] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2811.179229] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2811.181710] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2811.231269] Lustre: lustre-OST0001: new disk, initializing [ 2811.233727] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2812.561301] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2816.105810] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2816.490724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2816.495080] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2817.455178] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 2817.521590] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2818.809823] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 2818.873949] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2821.602836] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2821.604872] LustreError: Skipped 1 previous similar message [ 2827.307983] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 2827.375139] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2828.692223] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 2828.762489] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2852.087731] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2853.358696] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2855.628439] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2856.126288] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2896.351251] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2897.673980] LustreError: 228852:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772307587 with bad export cookie 3038846769484763530 [ 2897.676928] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2897.677734] LustreError: 228852:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2906.571726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2908.378489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2909.753451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2910.935612] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2913.397127] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 14:40:04 (1772307604) [ 2915.650484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2915.734936] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2915.744473] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2915.808673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2915.810577] Lustre: Skipped 5 previous similar messages [ 2917.244701] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2920.169187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2920.198649] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2921.577347] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2923.825946] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2925.563228] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2927.805184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2927.866685] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2929.416522] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2931.804935] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2941.309609] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2941.335177] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2942.766488] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2945.043540] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2948.585420] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 2949.848611] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 2949.909672] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2951.189070] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 2951.258942] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2953.025150] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff970d915a5800.ost_server_uuid 40 [ 2953.492789] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff970d915a5800.ost_server_uuid in FULL state after 0 sec [ 2953.798433] Lustre: Permanently deactivating lustre-OST0001 [ 2953.803035] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 2953.804626] Lustre: Skipped 1 previous similar message [ 2954.541203] Lustre: Permanently reactivating lustre-OST0001 [ 2957.794925] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2957.795068] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2957.798898] Lustre: Skipped 7 previous similar messages [ 2957.800821] LustreError: Skipped 2 previous similar messages [ 2957.801255] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2957.806386] Lustre: Skipped 22 previous similar messages [ 2961.514432] Lustre: server umount lustre-OST0001 complete [ 2961.515620] Lustre: Skipped 7 previous similar messages [ 2962.913783] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2962.918858] LustreError: Skipped 22 previous similar messages [ 2977.249313] LustreError: 237775:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772307667 with bad export cookie 3038846769484764776 [ 2977.251928] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2977.253817] LustreError: 237775:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2979.049776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2980.180855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2981.319569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2982.465364] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2984.445681] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 14:41:15 (1772307675) [ 2984.760692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2986.724489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2986.810686] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2986.820717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2986.822518] Lustre: Skipped 5 previous similar messages [ 2988.087503] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2990.609567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2990.637551] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2991.972820] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 2994.184261] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2995.794810] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2997.726440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2997.745925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2997.813435] Lustre: Permanently deactivating lustre-OST0000 [ 2999.088683] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3001.326246] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3003.376880] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3003.381148] LustreError: 247544:0:(osp_precreate.c:677:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 3003.384075] LustreError: 247544:0:(osp_precreate.c:1340:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 3003.422087] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3004.726301] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3006.991136] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3009.039514] Lustre: lustre-OST0001: deleting orphan objects from 0x0:35 to 0x0:65 [ 3012.810225] Lustre: Permanently reactivating lustre-OST0000 [ 3018.219542] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3018.221563] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3018.224461] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 3018.225490] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:130 to 0x280000400:161 [ 3018.235652] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3018.243073] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 3039.277934] LustreError: 244614:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772307729 with bad export cookie 3038846769484766358 [ 3039.279581] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3039.280809] LustreError: 244614:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3043.590452] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 3044.483450] Key type lgssc unregistered [ 3044.587343] LNet: 249551:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3044.589697] LNet: Removed LNI 192.168.204.147@tcp [ 3044.830782] Key type .llcrypt unregistered [ 3044.831678] Key type ._llcrypt unregistered [ 3052.336974] alg: No test for adler32 (adler32-zlib) [ 3053.088348] Key type ._llcrypt registered [ 3053.089407] Key type .llcrypt registered [ 3053.121875] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 3053.542205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3053.649784] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 3053.716349] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 3053.718211] LNet: Accept secure, port 988 [ 3055.287073] Key type lgssc registered [ 3055.570628] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3056.575322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3058.574869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3059.714226] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3059.730753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3060.937740] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3063.446171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3063.468482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3063.496625] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3063.505148] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3063.507458] Lustre: Permanently deactivating lustre-MDT0001 [ 3063.603827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3064.848553] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3065.322132] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3067.086276] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3068.802851] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3070.779866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3070.880149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3072.089166] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3073.953430] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3073.956453] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 3074.308489] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3076.371345] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3076.429531] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3077.701607] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3078.503643] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:67 to 0x2c0000400:97 [ 3078.505192] Lustre: lustre-OST0001: deleting orphan objects from 0x0:35 to 0x0:97 [ 3079.913873] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3081.422060] Lustre: Permanently reactivating lustre-MDT0001 [ 3081.427326] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 3081.429033] Lustre: Skipped 3 previous similar messages [ 3089.745820] Lustre: lustre-MDT0001: Client a24aa3b7-e93d-4e23-9e42-d30a62889470 (at 192.168.204.47@tcp) reconnecting [ 3089.911746] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3089.915397] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 3089.917986] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 3089.921791] LustreError: 254657:0:(llog_osd.c:2133:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 3089.922329] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to (at 0@lo) [ 3089.924145] LustreError: 254657:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 3089.928070] LustreError: 254657:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.204.147@tcp: cfg command failed: rc = -108 [ 3089.930259] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 3089.930259] [ 3089.932263] LustreError: 251053:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 3092.317066] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3092.795752] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3094.331188] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 3094.398366] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3095.692492] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 3095.764071] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3096.455841] Lustre: Permanently deactivating lustre-MDT0001 [ 3096.460512] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 3096.462230] Lustre: Skipped 2 previous similar messages [ 3098.038632] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3106.272397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3106.274822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3106.276764] Lustre: Skipped 3 previous similar messages [ 3106.278280] Lustre: Skipped 2 previous similar messages [ 3110.981645] Lustre: server umount lustre-MDT0000 complete [ 3111.392040] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3111.395283] LustreError: Skipped 3 previous similar messages [ 3112.183044] LustreError: 251850:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772307802 with bad export cookie 16984493027299832744 [ 3112.185394] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3112.187058] LustreError: 251850:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3112.308935] Lustre: server umount lustre-MDT0001 complete [ 3119.527167] Lustre: server umount lustre-OST0000 complete [ 3122.820853] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 14:43:33 (1772307813) [ 3126.051940] Lustre: server umount lustre-OST0001 complete [ 3128.034974] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 3130.482381] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 3133.377080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3135.287543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3137.056550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3138.890719] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3140.841431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3140.862731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3140.934903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3140.936656] Lustre: Skipped 2 previous similar messages [ 3140.944464] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3140.980420] Lustre: lustre-MDT0000: new disk, initializing [ 3141.001757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3141.007336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3142.213903] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3145.932439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3145.955899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3145.985680] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3145.998930] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3146.000532] Lustre: Skipped 1 previous similar message [ 3146.038118] Lustre: lustre-MDT0001: new disk, initializing [ 3146.071401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3146.076399] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3147.375600] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3150.856645] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3152.541633] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3154.536179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3154.561377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3154.563730] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3154.648525] Lustre: lustre-OST0000: new disk, initializing [ 3154.650333] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3154.674185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3154.677020] Lustre: Skipped 1 previous similar message [ 3155.572391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3155.575095] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3155.909581] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3159.315316] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3159.879118] LustreError: 262291:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3163.867946] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3163.897570] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3163.899696] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3163.942364] Lustre: lustre-OST0001: new disk, initializing [ 3163.943946] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3165.212071] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3168.690813] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3169.164891] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3169.967059] LustreError: 262291:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 3169.973283] LustreError: 262291:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3173.881831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3173.884608] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3180.063056] LustreError: 262291:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 3182.931176] Lustre: lustre-OST0001: Not available for connect from 192.168.204.47@tcp (stopping) [ 3184.099135] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3184.099237] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3184.101294] LustreError: Skipped 1 previous similar message [ 3184.106541] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3184.108775] Lustre: Skipped 1 previous similar message [ 3188.044651] Lustre: lustre-OST0001: Not available for connect from 192.168.204.47@tcp (stopping) [ 3193.165062] Lustre: lustre-OST0001: Not available for connect from 192.168.204.47@tcp (stopping) [ 3193.167186] Lustre: Skipped 2 previous similar messages [ 3195.359173] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3195.373483] Lustre: server umount lustre-OST0001 complete [ 3199.458887] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3199.462360] LustreError: Skipped 1 previous similar message [ 3204.577977] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3204.582268] LustreError: Skipped 1 previous similar message [ 3209.697341] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3209.698617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3209.700148] Lustre: Skipped 2 previous similar messages [ 3209.701510] Lustre: Skipped 2 previous similar messages [ 3213.267727] Lustre: server umount lustre-OST0000 complete [ 3214.816200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3214.819410] Lustre: Skipped 1 previous similar message [ 3222.044269] LustreError: 259525:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772307912 with bad export cookie 16984493027299835292 [ 3222.048347] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3222.050879] LustreError: 259525:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3236.319168] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3236.455314] Lustre: server umount lustre-MDT0001 complete [ 3236.457649] Lustre: Skipped 1 previous similar message [ 3242.160392] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 3243.600560] Key type lgssc unregistered [ 3243.734510] LNet: 264750:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3243.737898] LNet: Removed LNI 192.168.204.147@tcp [ 3244.174489] Key type .llcrypt unregistered [ 3244.177094] Key type ._llcrypt unregistered [ 3259.137518] alg: No test for adler32 (adler32-zlib) [ 3259.889282] Key type ._llcrypt registered [ 3259.890570] Key type .llcrypt registered [ 3259.934102] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 3267.510072] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 3268.068812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3268.194070] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 3268.270681] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 3268.272183] LNet: Accept secure, port 988 [ 3269.855116] Key type lgssc registered [ 3270.448475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3274.944969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3278.225305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3280.570046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3282.637046] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3285.266525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3285.297532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3286.416082] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3286.432448] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3286.481826] Lustre: lustre-MDT0000: new disk, initializing [ 3286.511919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3286.520589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3288.679926] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3293.826556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3293.874600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3293.930516] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3293.963077] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3293.966913] Lustre: Skipped 1 previous similar message [ 3294.017519] Lustre: lustre-MDT0001: new disk, initializing [ 3294.067696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3294.087242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3294.091530] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3296.094926] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3301.019444] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3303.704332] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3306.848819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3306.890954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3306.894989] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3307.029767] Lustre: lustre-OST0000: new disk, initializing [ 3307.033853] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3307.078548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3309.110844] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3309.775770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3309.783604] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3313.952114] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3316.181116] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3316.315107] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3318.275395] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3318.371714] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3320.294894] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3320.295244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3320.300109] LustreError: Skipped 1 previous similar message [ 3320.309563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3320.315928] Lustre: Skipped 1 previous similar message [ 3325.380782] Lustre: server umount lustre-OST0000 complete [ 3330.016385] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3330.020919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3330.027610] Lustre: Skipped 1 previous similar message [ 3330.030697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3333.478286] Lustre: server umount lustre-MDT0000 complete [ 3335.255984] LustreError: 268836:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772308025 with bad export cookie 13174620904755140439 [ 3335.262089] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3335.263903] LustreError: 268836:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3335.427829] Lustre: server umount lustre-MDT0001 complete [ 3339.063188] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 14:47:09 (1772308029) [ 3341.619476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3341.757504] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3341.777482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3343.556741] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3346.912230] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3347.086538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3347.230161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3349.353123] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3352.639299] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3354.716057] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3357.655636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3357.819552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3359.574413] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3362.589894] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3380.853707] Lustre: server umount lustre-OST0000 complete [ 3382.765097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3388.184464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3388.354794] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3388.360931] Lustre: Skipped 2 previous similar messages [ 3388.364111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3389.407176] Lustre: 267065:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772308072/real 1772308072] req@0000000029c60f2b x1858399589394688/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1772308079 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 3389.413463] Lustre: 267065:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 3390.379823] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3393.511542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3393.520266] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3393.520711] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 3393.523469] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 3393.533919] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 3393.536969] Lustre: Skipped 1 previous similar message [ 3393.853265] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3398.625960] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3398.627635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3398.632983] LustreError: Skipped 1 previous similar message [ 3398.641550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3398.650248] Lustre: Skipped 2 previous similar messages [ 3399.741261] Lustre: server umount lustre-OST0000 complete [ 3403.744881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3403.744965] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3403.750419] Lustre: Skipped 1 previous similar message [ 3403.750948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3403.756009] Lustre: Skipped 1 previous similar message [ 3408.458429] LustreError: 278023:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772308098 with bad export cookie 13174620904755141293 [ 3408.462359] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3408.466570] LustreError: 278023:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3408.592059] Lustre: server umount lustre-MDT0001 complete [ 3408.593220] Lustre: Skipped 1 previous similar message [ 3412.849984] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 3413.738108] Key type lgssc unregistered [ 3413.837356] LNet: 278826:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3413.839868] LNet: Removed LNI 192.168.204.147@tcp [ 3414.235152] Key type .llcrypt unregistered [ 3414.237204] Key type ._llcrypt unregistered [ 3424.486232] alg: No test for adler32 (adler32-zlib) [ 3425.238301] Key type ._llcrypt registered [ 3425.239564] Key type .llcrypt registered [ 3425.273641] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 3425.549539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3425.778657] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 3425.844952] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 3425.846469] LNet: Accept secure, port 988 [ 3427.423153] Key type lgssc registered [ 3427.795933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3431.079508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3432.249091] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3432.262528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3433.809510] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3437.434847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3437.535941] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3437.590507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3439.202921] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3442.376107] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3444.747643] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3447.533014] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3447.634756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3448.677988] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:97 [ 3448.862292] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3451.216978] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3468.260835] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3468.269639] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3468.269868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3468.278539] Lustre: Skipped 1 previous similar message [ 3471.218795] Lustre: server umount lustre-OST0000 complete [ 3473.378217] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3473.378346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3473.379333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3473.394630] Lustre: Skipped 2 previous similar messages [ 3478.496648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3478.500304] Lustre: Skipped 2 previous similar messages [ 3483.616233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3483.618808] Lustre: Skipped 1 previous similar message [ 3487.199198] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3487.264960] Lustre: server umount lustre-MDT0000 complete [ 3488.391970] LustreError: 280267:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772308178 with bad export cookie 5395025404422708805 [ 3488.394676] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3488.395967] LustreError: 280267:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3488.511412] Lustre: server umount lustre-MDT0001 complete [ 3493.590853] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 3494.814231] Key type lgssc unregistered [ 3494.949418] LNet: 284707:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3494.953820] LNet: Removed LNI 192.168.204.147@tcp [ 3495.348092] Key type .llcrypt unregistered [ 3495.350539] Key type ._llcrypt unregistered [ 3502.824654] alg: No test for adler32 (adler32-zlib) [ 3503.576344] Key type ._llcrypt registered [ 3503.577674] Key type .llcrypt registered [ 3503.611252] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 3503.966501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3504.126352] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 3504.193175] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 3504.194812] LNet: Accept secure, port 988 [ 3505.767166] Key type lgssc registered [ 3506.171362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3509.267442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3510.431270] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3510.446913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3512.027134] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3515.874194] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3515.899774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3516.005749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3517.994072] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3520.971249] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3523.116655] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3525.886068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3526.044456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3527.077563] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 3527.791733] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3531.043076] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3536.355308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3536.356373] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3536.356863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3536.362478] Lustre: Skipped 1 previous similar message [ 3541.473782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3541.475880] Lustre: Skipped 2 previous similar messages [ 3542.113476] Lustre: server umount lustre-OST0000 complete [ 3546.593312] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3546.593422] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3546.594064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3546.605111] Lustre: Skipped 1 previous similar message [ 3549.725175] Lustre: server umount lustre-MDT0000 complete [ 3551.446042] LustreError: 286763:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772308241 with bad export cookie 9281087600955603831 [ 3551.448822] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3551.451925] LustreError: 286763:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3551.583875] Lustre: server umount lustre-MDT0001 complete [ 3557.116587] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 3558.307605] Key type lgssc unregistered [ 3558.428454] LNet: 289798:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3558.431766] LNet: Removed LNI 192.168.204.147@tcp [ 3558.796473] Key type .llcrypt unregistered [ 3558.797981] Key type ._llcrypt unregistered [ 3566.647696] alg: No test for adler32 (adler32-zlib) [ 3567.399341] Key type ._llcrypt registered [ 3567.400329] Key type .llcrypt registered [ 3567.441499] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 3567.772719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3567.972987] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 3568.042513] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 3568.044065] LNet: Accept secure, port 988 [ 3569.623075] Key type lgssc registered [ 3570.038319] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3573.480780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3574.649236] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3574.663968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3576.125418] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3579.840513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3579.871969] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3579.948076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3581.606419] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3585.233438] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3587.838600] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3590.638536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3590.816767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3592.584294] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3595.358829] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3605.166675] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 3610.596337] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3610.597990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3610.600729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3610.612069] Lustre: Skipped 1 previous similar message [ 3614.751504] Lustre: server umount lustre-OST0000 complete [ 3620.833530] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3620.833784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3620.834643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3620.834652] Lustre: Skipped 1 previous similar message [ 3620.838665] LustreError: Skipped 1 previous similar message [ 3620.847982] Lustre: Skipped 1 previous similar message [ 3622.363348] Lustre: server umount lustre-MDT0000 complete [ 3624.181554] LustreError: 291044:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772308314 with bad export cookie 4701406845175128160 [ 3624.186925] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3624.189181] LustreError: 291044:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3624.341716] Lustre: server umount lustre-MDT0001 complete [ 3630.257382] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 3631.727358] Key type lgssc unregistered [ 3631.874632] LNet: 294736:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3631.880028] LNet: Removed LNI 192.168.204.147@tcp [ 3632.320417] Key type .llcrypt unregistered [ 3632.323320] Key type ._llcrypt unregistered [ 3643.170236] alg: No test for adler32 (adler32-zlib) [ 3643.922349] Key type ._llcrypt registered [ 3643.923467] Key type .llcrypt registered [ 3643.971101] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 3644.382771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3644.517816] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 3644.590287] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 3644.592238] LNet: Accept secure, port 988 [ 3646.167099] Key type lgssc registered [ 3646.636019] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3649.475635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3650.641597] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3650.655426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3651.990974] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3654.655288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3654.770564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3656.103139] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3658.408162] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3660.140391] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3662.167866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3662.259850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3663.557558] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3665.319797] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 3666.431028] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3680.737134] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3680.742239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3680.751642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3684.835927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3684.836254] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3684.847506] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3686.780366] Lustre: server umount lustre-OST0000 complete [ 3689.954505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3689.955604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3689.961917] Lustre: Skipped 1 previous similar message [ 3689.968023] Lustre: Skipped 2 previous similar messages [ 3694.622057] Lustre: server umount lustre-MDT0000 complete [ 3695.073436] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3696.398978] LustreError: 296177:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772308386 with bad export cookie 13032583048911862209 [ 3696.406040] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3696.406474] LustreError: 296177:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3696.595195] Lustre: server umount lustre-MDT0001 complete [ 3701.681138] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 3702.723402] Key type lgssc unregistered [ 3702.842444] LNet: 300668:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3702.845141] LNet: Removed LNI 192.168.204.147@tcp [ 3703.186175] Key type .llcrypt unregistered [ 3703.187197] Key type ._llcrypt unregistered [ 3710.578423] alg: No test for adler32 (adler32-zlib) [ 3711.329437] Key type ._llcrypt registered [ 3711.330416] Key type .llcrypt registered [ 3711.377035] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 3711.712937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3711.973721] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 3712.046430] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 3712.047896] LNet: Accept secure, port 988 [ 3713.623107] Key type lgssc registered [ 3714.016889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3717.181236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3718.348254] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3718.365447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3720.349198] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3723.745955] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3724.410252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3724.594859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3726.530136] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3730.016228] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3732.735897] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3736.148909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3736.334890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3738.150305] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3741.334645] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3748.654654] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 3758.048914] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3758.051277] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3758.057048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3759.072506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3759.074212] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3762.199100] Lustre: server umount lustre-OST0000 complete [ 3764.193636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3764.194810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3764.200880] Lustre: Skipped 1 previous similar message [ 3764.207097] Lustre: Skipped 2 previous similar messages [ 3769.312587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3769.317147] Lustre: Skipped 1 previous similar message [ 3774.431879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3774.433423] Lustre: Skipped 1 previous similar message [ 3778.015148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3778.089157] Lustre: server umount lustre-MDT0000 complete [ 3779.613562] LustreError: 301912:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772308469 with bad export cookie 5866354182065935119 [ 3779.616984] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3779.620190] LustreError: 301912:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3779.764445] Lustre: server umount lustre-MDT0001 complete [ 3784.799836] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 3786.070210] Key type lgssc unregistered [ 3786.182442] LNet: 305789:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3786.185511] LNet: Removed LNI 192.168.204.147@tcp [ 3786.558251] Key type .llcrypt unregistered [ 3786.559479] Key type ._llcrypt unregistered [ 3793.592494] alg: No test for adler32 (adler32-zlib) [ 3794.344369] Key type ._llcrypt registered [ 3794.345331] Key type .llcrypt registered [ 3794.389650] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 3794.865370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3794.974728] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 3795.047907] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 3795.049649] LNet: Accept secure, port 988 [ 3796.631140] Key type lgssc registered [ 3797.125929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3800.693189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3801.868579] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3801.886089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3803.792241] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3807.201359] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3807.957591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3808.146647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3809.943911] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3813.223214] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3815.997130] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3819.039132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3819.200184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3821.142679] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3823.583226] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3832.550087] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:257 [ 3837.921972] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3837.923741] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3837.925275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3837.940219] Lustre: Skipped 1 previous similar message [ 3839.159770] Lustre: server umount lustre-OST0000 complete [ 3843.040118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3843.040686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3843.043313] Lustre: Skipped 1 previous similar message [ 3843.044865] Lustre: Skipped 1 previous similar message [ 3848.160627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3848.164022] Lustre: Skipped 2 previous similar messages [ 3853.280083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3853.282436] Lustre: Skipped 1 previous similar message [ 3855.327087] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3855.367322] Lustre: server umount lustre-MDT0000 complete [ 3856.957943] LustreError: 307037:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772308547 with bad export cookie 14603213838546601704 [ 3856.963098] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3856.968086] LustreError: 307037:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3857.096460] Lustre: server umount lustre-MDT0001 complete [ 3862.960599] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 3864.522306] Key type lgssc unregistered [ 3864.689554] LNet: 310729:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3864.694233] LNet: Removed LNI 192.168.204.147@tcp [ 3865.168997] Key type .llcrypt unregistered [ 3865.171508] Key type ._llcrypt unregistered [ 3870.934731] alg: No test for adler32 (adler32-zlib) [ 3871.689447] Key type ._llcrypt registered [ 3871.690658] Key type .llcrypt registered [ 3871.738874] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 3879.587905] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 3885.611601] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 3886.217622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3886.378880] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 3886.473368] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 3886.475168] LNet: Accept secure, port 988 [ 3888.063168] Key type lgssc registered [ 3888.594255] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3892.647152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3895.713323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3898.710498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3901.681532] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3904.960162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3904.996504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3906.112686] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3906.122505] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3906.163412] Lustre: lustre-MDT0000: new disk, initializing [ 3906.190414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3906.199133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3907.923529] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3912.767759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3912.797361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3912.833175] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3912.855812] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3912.858639] Lustre: Skipped 1 previous similar message [ 3912.895867] Lustre: lustre-MDT0001: new disk, initializing [ 3912.941975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3912.958163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3912.962217] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3914.769410] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3919.536833] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3921.635229] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3924.389560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3924.432095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3924.436517] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3924.566232] Lustre: lustre-OST0000: new disk, initializing [ 3924.568307] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3924.608132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3926.549235] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3931.512947] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3932.158032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3932.162862] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3933.643283] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3933.764291] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3935.812403] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3935.934610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3937.249869] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3937.250897] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3937.254600] LustreError: Skipped 1 previous similar message [ 3937.255738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3937.265414] Lustre: Skipped 1 previous similar message [ 3937.273970] Lustre: Skipped 1 previous similar message [ 3938.861922] Lustre: server umount lustre-OST0000 complete [ 3942.371539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3942.374667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3942.379020] Lustre: Skipped 1 previous similar message [ 3946.852589] Lustre: server umount lustre-MDT0000 complete [ 3947.489167] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3947.497703] LustreError: Skipped 1 previous similar message [ 3948.752841] LustreError: 314469:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772308638 with bad export cookie 16963906134168306667 [ 3948.760291] LustreError: 314469:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3948.760727] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3948.927003] Lustre: server umount lustre-MDT0001 complete [ 3952.833725] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 14:57:23 (1772308643) [ 3955.772126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3955.952337] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3955.984460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3958.127600] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3961.313648] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3961.898432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3962.090297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3964.097945] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3967.081631] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3969.598339] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3972.799805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3972.985186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3975.009390] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 3978.481568] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3996.646043] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3996.646479] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3996.647902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3996.647915] Lustre: Skipped 1 previous similar message [ 3996.664580] Lustre: Skipped 1 previous similar message [ 3998.747526] Lustre: server umount lustre-OST0000 complete [ 4001.761747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4001.762824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4001.769603] Lustre: Skipped 1 previous similar message [ 4001.776062] Lustre: Skipped 2 previous similar messages [ 4006.496668] Lustre: server umount lustre-MDT0000 complete [ 4006.881930] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4006.895680] LustreError: Skipped 1 previous similar message [ 4008.289758] LustreError: 319006:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772308698 with bad export cookie 16963906134168307507 [ 4008.295953] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4008.296922] LustreError: 319006:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4013.857597] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 4015.291915] Key type lgssc unregistered [ 4015.438457] LNet: 322701:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4015.444039] LNet: Removed LNI 192.168.204.147@tcp [ 4015.913582] Key type .llcrypt unregistered [ 4015.915797] Key type ._llcrypt unregistered [ 4027.679519] alg: No test for adler32 (adler32-zlib) [ 4028.430349] Key type ._llcrypt registered [ 4028.431324] Key type .llcrypt registered [ 4028.470213] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 4035.106516] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 4035.416751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4035.615991] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 4035.682803] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 4035.684515] LNet: Accept secure, port 988 [ 4037.263060] Key type lgssc registered [ 4037.647210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4041.482913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4044.450839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4047.291045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4050.120542] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4052.634908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4055.196683] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4058.489062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4058.527754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4059.626189] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4059.636464] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4059.676922] Lustre: lustre-MDT0000: new disk, initializing [ 4059.702377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4059.710130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4061.335512] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4066.631290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4066.671149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4066.723206] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4066.747129] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4066.750806] Lustre: Skipped 1 previous similar message [ 4066.794520] Lustre: lustre-MDT0001: new disk, initializing [ 4066.834184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4066.851905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4066.858276] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4068.809270] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4073.265819] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4075.021138] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4077.562735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4077.593709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4077.599587] Lustre: lustre-OST2710-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4077.732489] Lustre: lustre-OST2710: new disk, initializing [ 4077.734658] Lustre: srv-lustre-OST2710: No data found on store. Initialize space: rc = -61 [ 4077.758257] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4079.745970] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4083.214234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4083.218505] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4084.657045] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4087.571805] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4087.651155] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4087.659930] Lustre: lustre-OST03e8-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4087.836814] Lustre: lustre-OST03e8: new disk, initializing [ 4087.841988] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space: rc = -61 [ 4087.930942] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4090.767581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4090.777905] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4091.750820] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4097.020440] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4102.055839] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 40 [ 4102.307907] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4106.267246] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 40 [ 4106.518097] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4110.572686] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 40 [ 4110.783589] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4114.083737] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 40 [ 4114.263932] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4116.448322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4116.451282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4116.453755] Lustre: Skipped 3 previous similar messages [ 4116.456505] Lustre: Skipped 2 previous similar messages [ 4121.569041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4121.571897] Lustre: Skipped 3 previous similar messages [ 4122.374284] Lustre: server umount lustre-MDT0000 complete [ 4123.726949] LustreError: 326635:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772308813 with bad export cookie 2923058015244131030 [ 4123.729946] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4123.730766] LustreError: 326635:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4123.854867] Lustre: server umount lustre-MDT0001 complete [ 4131.295130] Lustre: 332123:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772308815/real 1772308815] req@0000000064d7ac84 x1858400393651840/t0(0) o39->lustre-MDT0001-lwp-OST2710@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772308821 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4131.319716] Lustre: server umount lustre-OST2710 complete [ 4137.951156] Lustre: 332324:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772308822/real 1772308822] req@000000000236771a x1858400393652416/t0(0) o39->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772308828 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4138.041127] Lustre: server umount lustre-OST03e8 complete [ 4143.787778] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 4146.761132] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 4150.188190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4152.534773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4154.591947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4156.708055] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4159.052664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4159.080641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4159.163392] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4159.173275] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4159.212036] Lustre: lustre-MDT0000: new disk, initializing [ 4159.252563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4159.262100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4160.740695] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4164.773834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4164.794251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4164.823790] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4164.916455] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4166.306761] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4169.966676] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4171.944488] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4174.167614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4174.189250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4174.192155] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4174.269899] Lustre: lustre-OST0000: new disk, initializing [ 4174.271854] Lustre: Skipped 1 previous similar message [ 4174.273864] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4174.276840] Lustre: Skipped 2 previous similar messages [ 4174.300018] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4174.302202] Lustre: Skipped 1 previous similar message [ 4175.579514] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4175.604451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4175.608302] Lustre: Skipped 1 previous similar message [ 4175.610300] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4179.253859] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4180.706493] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4180.791527] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4182.145453] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4182.222496] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4186.080147] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4186.081117] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4186.082188] LustreError: Skipped 1 previous similar message [ 4186.085246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4186.087132] Lustre: Skipped 1 previous similar message [ 4191.201365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4191.203535] Lustre: Skipped 1 previous similar message [ 4196.320635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4196.322413] Lustre: Skipped 1 previous similar message [ 4197.343105] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4197.396648] Lustre: server umount lustre-OST0000 complete [ 4200.927788] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4200.930461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4200.933591] Lustre: Skipped 1 previous similar message [ 4205.999336] LustreError: 336081:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772308896 with bad export cookie 2923058015244132633 [ 4206.000182] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4206.003785] LustreError: 336081:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4206.123488] Lustre: server umount lustre-MDT0001 complete [ 4206.125512] Lustre: Skipped 1 previous similar message [ 4208.947261] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 15:01:39 (1772308899) [ 4209.488681] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4210.088210] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 15:01:40 (1772308900) [ 4211.298796] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 4214.045389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4215.362989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4216.785654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4218.093216] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4218.450976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4220.675373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4220.760819] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4220.771062] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4220.825236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4222.236635] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4225.043625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4225.072151] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4226.557625] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4228.959956] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4230.881470] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4233.150520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4233.176030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4233.238203] Lustre: Denying initial registration attempt from nid 192.168.204.147@tcp, specified as failover [ 4233.241582] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4233.246660] LustreError: 343253:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -99 [ 4233.249996] LustreError: 343253:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 4233.252440] LustreError: 343253:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 4233.301090] Lustre: server umount lustre-OST0000 complete [ 4233.302621] LustreError: 343253:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4235.743842] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4235.746851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4235.750171] Lustre: Skipped 1 previous similar message [ 4235.751688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4235.753423] Lustre: Skipped 3 previous similar messages [ 4242.300183] LustreError: 342098:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772308932 with bad export cookie 2923058015244133473 [ 4242.302866] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4242.304326] LustreError: 342098:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4247.213769] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 4248.155310] Key type lgssc unregistered [ 4248.268294] LNet: 344327:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4248.270522] LNet: Removed LNI 192.168.204.147@tcp [ 4248.556125] Key type .llcrypt unregistered [ 4248.557223] Key type ._llcrypt unregistered [ 4253.506778] alg: No test for adler32 (adler32-zlib) [ 4254.258574] Key type ._llcrypt registered [ 4254.259957] Key type .llcrypt registered [ 4254.303879] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 4254.718073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4255.015555] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 4255.096276] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 4255.098431] LNet: Accept secure, port 988 [ 4256.679132] Key type lgssc registered [ 4257.060915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4259.508777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4260.763589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4261.967610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4263.196628] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4263.527375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4268.977367] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 4272.169089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4273.294015] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4273.305206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4273.377286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4274.898709] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4277.770469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4277.800353] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4277.810088] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4277.893796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4279.300026] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4281.758479] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4283.606799] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4285.698940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4285.725271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4285.790124] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4285.839304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4287.225564] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4289.505328] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4293.600900] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4293.601340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4293.602440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4293.603230] LustreError: Skipped 1 previous similar message [ 4293.606445] Lustre: Skipped 1 previous similar message [ 4296.297219] Lustre: server umount lustre-OST0000 complete [ 4298.720575] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4298.720962] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4298.725505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4298.729903] Lustre: Skipped 2 previous similar messages [ 4303.618591] Lustre: server umount lustre-MDT0000 complete [ 4303.840196] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4303.844772] LustreError: Skipped 1 previous similar message [ 4304.807091] LustreError: 347192:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772308994 with bad export cookie 4153649554490864651 [ 4304.808903] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4304.810728] LustreError: 347192:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4304.938986] Lustre: server umount lustre-MDT0001 complete [ 4308.293449] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 4309.229147] Key type lgssc unregistered [ 4309.331347] LNet: 350891:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4309.333653] LNet: Removed LNI 192.168.204.147@tcp [ 4309.605372] Key type .llcrypt unregistered [ 4309.606983] Key type ._llcrypt unregistered [ 4318.117435] alg: No test for adler32 (adler32-zlib) [ 4318.869316] Key type ._llcrypt registered [ 4318.870683] Key type .llcrypt registered [ 4318.908407] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 4319.323895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4319.456665] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 4319.533316] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 4319.535485] LNet: Accept secure, port 988 [ 4321.119144] Key type lgssc registered [ 4321.460174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4323.948626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4325.106125] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4325.120742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4326.432396] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4329.077823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4329.180777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4330.582771] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4332.906520] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4334.723645] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4336.874287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4336.973056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4338.322243] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4340.762961] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4349.409151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4349.410129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4349.412359] Lustre: Skipped 2 previous similar messages [ 4349.414073] Lustre: Skipped 1 previous similar message [ 4352.339897] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.47@tcp (stopping) [ 4352.343329] Lustre: Skipped 1 previous similar message [ 4354.528459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4354.532888] Lustre: Skipped 2 previous similar messages [ 4355.079843] Lustre: server umount lustre-MDT0000 complete [ 4356.454288] LustreError: 354299:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772309046 with bad export cookie 10179015248530974410 [ 4356.455815] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4356.458403] LustreError: 354299:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4357.452601] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4357.462236] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.47@tcp (stopping) [ 4359.647929] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4359.648138] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4361.695854] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4361.695922] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4361.701513] Lustre: Skipped 1 previous similar message [ 4362.611816] Lustre: server umount lustre-MDT0001 complete [ 4364.858088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4367.831389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4372.960513] Lustre: Evicted from MGS (at 192.168.204.147@tcp) after server handle changed from 0x8d43207013f926ca to 0x8d43207013f942a7 [ 4372.965586] Lustre: MGC192.168.204.147@tcp: Connection restored to 192.168.204.147@tcp (at 0@lo) [ 4373.065844] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4373.093823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4373.119842] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:193 [ 4374.724156] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4377.670959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4377.812348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4378.852105] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 4379.360236] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4381.999853] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4382.601178] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4384.587880] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4387.042823] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 4390.270203] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 5 sec [ 4392.417421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4392.421913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4392.423759] Lustre: Skipped 1 previous similar message [ 4397.596935] Lustre: server umount lustre-OST0000 complete [ 4402.656133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4402.659349] Lustre: Skipped 1 previous similar message [ 4412.895874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4412.898426] Lustre: Skipped 7 previous similar messages [ 4413.407084] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4413.448133] Lustre: server umount lustre-MDT0000 complete [ 4414.655912] LustreError: 352331:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772309104 with bad export cookie 10179015248530981543 [ 4414.658426] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4414.660398] LustreError: 352331:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4419.173967] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 4420.126514] Key type lgssc unregistered [ 4420.238260] LNet: 358842:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4420.240662] LNet: Removed LNI 192.168.204.147@tcp [ 4420.535258] Key type .llcrypt unregistered [ 4420.536404] Key type ._llcrypt unregistered [ 4429.175275] alg: No test for adler32 (adler32-zlib) [ 4429.926351] Key type ._llcrypt registered [ 4429.927725] Key type .llcrypt registered [ 4429.964347] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 4430.248431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4430.477191] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 4430.544530] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 4430.545862] LNet: Accept secure, port 988 [ 4432.119137] Key type lgssc registered [ 4432.440465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4434.873352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4436.009994] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4436.023564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4437.463222] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4440.308105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4440.418444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4441.876080] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4444.400972] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4446.251567] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4450.784265] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4450.784366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4450.785092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4450.791799] Lustre: Skipped 1 previous similar message [ 4453.444315] Lustre: server umount lustre-MDT0000 complete [ 4454.714072] LustreError: 362025:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772309144 with bad export cookie 15367437483277217714 [ 4454.715647] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4454.719099] LustreError: 362025:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4454.833552] Lustre: server umount lustre-MDT0001 complete [ 4459.565084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4459.663395] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4459.674478] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4459.744058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4461.189738] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4464.169816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4464.202385] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4464.212104] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4464.296735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4465.745101] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4468.149831] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4470.032049] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4472.511021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4472.578823] LustreError: 13b-9: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4472.582528] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4472.586164] LustreError: 364989:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 4472.588386] LustreError: 364989:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 4472.590675] LustreError: 364989:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 4472.639356] Lustre: server umount lustre-OST0000 complete [ 4472.641321] LustreError: 364989:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4474.723210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4474.783077] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4474.845155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4476.160598] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4478.537485] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4479.977403] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:225 [ 4480.882740] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4480.908449] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4480.911294] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4480.934335] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4480.958509] Lustre: lustre-OST0001: new disk, initializing [ 4480.960028] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 4482.298282] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4484.772926] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4485.722949] Lustre: server umount lustre-OST0001 complete [ 4489.708629] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4489.710390] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4489.711392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4489.711397] Lustre: Skipped 1 previous similar message [ 4489.720458] Lustre: Skipped 1 previous similar message [ 4493.081785] Lustre: server umount lustre-OST0000 complete [ 4494.816175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4494.816810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4494.822395] Lustre: Skipped 1 previous similar message [ 4494.824632] Lustre: Skipped 1 previous similar message [ 4497.676993] LustreError: 366392:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772309187 with bad export cookie 15367437483277218253 [ 4497.679609] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4497.680156] LustreError: 366392:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4501.189688] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 4502.154962] Key type lgssc unregistered [ 4502.264440] LNet: 368354:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4502.267395] LNet: Removed LNI 192.168.204.147@tcp [ 4502.561672] Key type .llcrypt unregistered [ 4502.562640] Key type ._llcrypt unregistered [ 4513.407049] alg: No test for adler32 (adler32-zlib) [ 4514.158291] Key type ._llcrypt registered [ 4514.159311] Key type .llcrypt registered [ 4514.197624] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 4519.897068] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 4520.301729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4520.421788] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 4520.493927] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 4520.496221] LNet: Accept secure, port 988 [ 4522.071091] Key type lgssc registered [ 4522.354536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4524.949228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4526.929268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4528.741370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4530.617526] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4532.771967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4532.794734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4533.871577] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4533.881125] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4533.917329] Lustre: lustre-MDT0000: new disk, initializing [ 4533.938506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4533.946225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4535.293121] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4539.144067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4539.162703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4539.195161] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4539.218627] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4539.219582] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4539.223297] Lustre: Skipped 1 previous similar message [ 4539.254937] Lustre: lustre-MDT0001: new disk, initializing [ 4539.282416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4539.293062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4539.295794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4540.585605] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4544.203884] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4545.967556] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4548.013158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4548.040203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4548.042477] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4548.120044] Lustre: lustre-OST0000: new disk, initializing [ 4548.122045] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4548.144285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4549.447196] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4553.000100] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4553.718451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4553.724773] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4554.409762] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4554.473338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4555.795028] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4555.873211] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4558.817422] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4558.817553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4558.819582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4558.821314] LustreError: Skipped 1 previous similar message [ 4558.826253] Lustre: Skipped 1 previous similar message [ 4562.579688] Lustre: server umount lustre-OST0000 complete [ 4563.935901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4563.936299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4563.938872] Lustre: Skipped 1 previous similar message [ 4563.940307] Lustre: Skipped 1 previous similar message [ 4569.055874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4569.058117] Lustre: Skipped 2 previous similar messages [ 4569.796282] Lustre: server umount lustre-MDT0000 complete [ 4571.039067] LustreError: 372433:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772309261 with bad export cookie 6272559610761209305 [ 4571.041867] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4571.044547] LustreError: 372433:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4571.174767] Lustre: server umount lustre-MDT0001 complete [ 4573.899211] Lustre: DEBUG MARKER: == conf-sanity test 60: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 15:07:44 (1772309264) [ 4575.824800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4577.797328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4585.134380] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 4587.934178] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 4590.872677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4592.830077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4594.554349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4596.358174] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4598.387815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4598.408408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4598.481145] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4598.489983] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4598.524343] Lustre: lustre-MDT0000: new disk, initializing [ 4598.543918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4598.549222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4599.856804] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4603.741485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4603.760529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4603.789568] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4603.809765] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4603.812430] Lustre: Skipped 1 previous similar message [ 4603.849065] Lustre: lustre-MDT0001: new disk, initializing [ 4603.874872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4603.886298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4603.889076] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4605.123464] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4608.815219] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4610.616605] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4612.722215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4612.747309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4612.749377] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4612.829076] Lustre: lustre-OST0000: new disk, initializing [ 4612.830638] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4612.853429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4614.121737] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4617.647579] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4618.968896] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4620.272492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4620.276382] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4621.067798] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4622.367499] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4622.439024] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4625.377255] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4625.377757] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4625.378553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4625.379556] LustreError: Skipped 1 previous similar message [ 4625.384676] Lustre: Skipped 1 previous similar message [ 4629.139892] Lustre: server umount lustre-OST0000 complete [ 4630.496107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4630.499432] Lustre: Skipped 1 previous similar message [ 4635.615856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4635.617996] Lustre: Skipped 4 previous similar messages [ 4636.480448] Lustre: server umount lustre-MDT0000 complete [ 4637.669975] LustreError: 380232:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772309327 with bad export cookie 6272559610761210159 [ 4637.672182] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4637.672540] LustreError: 380232:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4640.264833] Lustre: DEBUG MARKER: == conf-sanity test 61: large xattr ====================== 15:08:51 (1772309331) [ 4642.797558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4642.923708] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4642.944198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4644.175927] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4646.680761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4647.976501] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4650.147080] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4651.748915] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4653.705041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4655.076381] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4657.245391] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4666.827738] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61.conf-sanity [ 4667.382407] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61.conf-sanity [ 4667.914582] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61.conf-sanity [ 4668.407639] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61.conf-sanity after remounting MDS [ 4669.002293] Lustre: Failing over lustre-MDT0000 [ 4669.129414] Lustre: server umount lustre-MDT0000 complete [ 4669.131326] Lustre: Skipped 1 previous similar message [ 4671.456783] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4671.457626] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4671.460624] Lustre: Skipped 1 previous similar message [ 4673.875364] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4673.879977] LustreError: Skipped 2 previous similar messages [ 4676.576206] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4676.580601] LustreError: Skipped 2 previous similar messages [ 4678.623105] Lustre: 370671:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772309361/real 1772309361] req@0000000060e798b4 x1858400902224000/t0(0) o400->MGC192.168.204.147@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772309368 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4678.628714] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4681.051633] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4681.056693] LustreError: Skipped 4 previous similar messages [ 4682.032278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4684.769937] Lustre: Evicted from MGS (at 192.168.204.147@tcp) after server handle changed from 0x570c9bbb87e26c77 to 0x570c9bbb87e2704b [ 4684.773728] LustreError: 388177:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.204.147@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (000000004a522992) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4684.779330] Lustre: MGC192.168.204.147@tcp: Connection restored to 192.168.204.147@tcp (at 0@lo) [ 4684.845526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4684.848101] Lustre: Skipped 2 previous similar messages [ 4684.871547] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4686.152835] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4686.156255] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4689.891230] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 4689.897420] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4689.914949] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 4691.807610] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4692.369670] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4694.156654] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61.conf-sanity [ 4694.763182] Lustre: Failing over lustre-MDT0000 [ 4694.881538] Lustre: server umount lustre-MDT0000 complete [ 4695.008922] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4695.009863] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4695.014960] Lustre: Skipped 3 previous similar messages [ 4695.019727] LustreError: Skipped 7 previous similar messages [ 4695.021180] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4702.175134] Lustre: 384850:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772309385/real 1772309385] req@000000003fedf5f2 x1858400902229696/t0(0) o101->MGC192.168.204.147@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1772309392 ref 2 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 4702.175212] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4702.183357] Lustre: 384850:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4702.189103] LustreError: 384850:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 4707.834988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4708.322106] Lustre: Evicted from MGS (at 192.168.204.147@tcp) after server handle changed from 0x570c9bbb87e2704b to 0x570c9bbb87e272c8 [ 4708.325100] Lustre: MGC192.168.204.147@tcp: Connection restored to (at 0@lo) [ 4708.326842] Lustre: Skipped 2 previous similar messages [ 4708.426259] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4708.684056] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4709.677298] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4713.446498] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 4713.451542] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4713.466596] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [ 4715.086795] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4715.546068] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4718.560772] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4718.562077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4718.566094] Lustre: Skipped 2 previous similar messages [ 4718.568099] Lustre: Skipped 1 previous similar message [ 4718.574348] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4725.207165] LustreError: 384840:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772309415 with bad export cookie 6272559610761212616 [ 4725.210260] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4725.210876] LustreError: 384840:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4731.871181] Lustre: 390630:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772309416/real 1772309416] req@00000000aada3ac0 x1858400902238656/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772309422 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4731.945116] Lustre: server umount lustre-OST0000 complete [ 4731.946441] Lustre: Skipped 2 previous similar messages [ 4736.992940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4737.140464] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4737.146666] LustreError: Skipped 15 previous similar messages [ 4738.574479] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4741.447470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4742.956465] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4745.413693] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4747.312544] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4749.374698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4749.464331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4749.465971] Lustre: Skipped 3 previous similar messages [ 4750.747061] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4752.550620] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [ 4753.221735] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4762.080413] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4762.082765] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4762.084365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4762.087211] Lustre: Skipped 1 previous similar message [ 4762.089603] Lustre: Skipped 4 previous similar messages [ 4772.320557] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4774.658069] LustreError: 392242:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772309464 with bad export cookie 6272559610761213330 [ 4774.660585] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4774.662420] LustreError: 392242:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4779.075960] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 4779.972777] Key type lgssc unregistered [ 4780.074282] LNet: 395108:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4780.077275] LNet: Removed LNI 192.168.204.147@tcp [ 4780.353108] Key type .llcrypt unregistered [ 4780.354670] Key type ._llcrypt unregistered [ 4788.683423] alg: No test for adler32 (adler32-zlib) [ 4789.434298] Key type ._llcrypt registered [ 4789.435426] Key type .llcrypt registered [ 4789.467327] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 4789.762367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4789.989163] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 4790.057419] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 4790.058993] LNet: Accept secure, port 988 [ 4791.639125] Key type lgssc registered [ 4791.971889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4794.491704] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4794.496420] LustreError: 396596:0:(osd_handler.c:8215:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 4794.499804] LustreError: 396596:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 4794.502517] LustreError: 396596:0:(obd_mount.c:200:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 4794.504580] LustreError: 396596:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 4794.507071] LustreError: 396596:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4796.885486] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4796.888660] LustreError: 396993:0:(osd_handler.c:8215:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 4796.892087] LustreError: 396993:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-22) [ 4796.894032] LustreError: 396993:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 4796.896114] LustreError: 396993:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 4796.899349] LustreError: 396993:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4802.702781] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 4803.711179] Key type lgssc unregistered [ 4803.818313] LNet: 397949:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4803.821808] LNet: Removed LNI 192.168.204.147@tcp [ 4804.125036] Key type .llcrypt unregistered [ 4804.126136] Key type ._llcrypt unregistered [ 4809.723139] alg: No test for adler32 (adler32-zlib) [ 4810.474413] Key type ._llcrypt registered [ 4810.476210] Key type .llcrypt registered [ 4810.509336] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 4816.161975] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 4816.518320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4816.760468] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 4816.846992] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 4816.848490] LNet: Accept secure, port 988 [ 4818.423083] Key type lgssc registered [ 4818.764948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4821.648420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4823.824882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4825.755666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4827.819266] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4830.065383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4830.087684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4831.177849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4831.188417] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4831.226817] Lustre: lustre-MDT0000: new disk, initializing [ 4831.249147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4831.255213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4832.651933] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4836.808365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4836.830667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4836.864722] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4836.882989] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4836.886693] Lustre: Skipped 1 previous similar message [ 4836.921479] Lustre: lustre-MDT0001: new disk, initializing [ 4836.950629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4836.962499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4836.965661] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4838.399533] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4842.243451] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4844.158893] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4846.347708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4846.371684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4846.375203] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4846.462697] Lustre: lustre-OST0000: new disk, initializing [ 4846.465578] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4846.471130] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 4846.507894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4847.895050] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4851.533405] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4851.686915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4851.691038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4852.943475] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4853.008902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4854.422605] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4854.496044] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4856.801850] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4856.802052] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4856.809217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4856.811366] Lustre: Skipped 1 previous similar message [ 4857.169251] Lustre: server umount lustre-OST0000 complete [ 4861.920485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4861.921071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4861.924071] Lustre: Skipped 2 previous similar messages [ 4861.926669] Lustre: Skipped 1 previous similar message [ 4864.578443] Lustre: server umount lustre-MDT0000 complete [ 4865.834925] LustreError: 402072:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772309555 with bad export cookie 11555121765330360745 [ 4865.838035] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4865.838806] LustreError: 402072:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4865.958458] Lustre: server umount lustre-MDT0001 complete [ 4868.733742] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 4869.283280] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 15:12:39 (1772309559) [ 4871.472547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4871.625870] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4871.647893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4873.088680] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4875.939096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4876.065760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4877.511495] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4880.143987] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4882.104928] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4884.281838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4884.386721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4885.775556] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4888.211583] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4890.443866] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4890.466599] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4890.471350] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4890.525032] Lustre: lustre-OST0001: new disk, initializing [ 4890.529623] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 4891.926500] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4895.668764] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4898.805572] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4903.526364] Lustre: server umount lustre-OST0001 complete [ 4903.907294] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4903.911108] LustreError: Skipped 1 previous similar message [ 4909.025525] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4909.029411] LustreError: Skipped 1 previous similar message [ 4914.146383] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4914.152903] LustreError: Skipped 1 previous similar message [ 4917.215993] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4917.220026] LustreError: Skipped 1 previous similar message [ 4917.221396] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4917.225058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4917.226640] Lustre: Skipped 2 previous similar messages [ 4918.240069] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4919.266508] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4919.270721] LustreError: Skipped 1 previous similar message [ 4921.687186] Lustre: server umount lustre-OST0000 complete [ 4924.384763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4924.388768] Lustre: Skipped 2 previous similar messages [ 4929.503913] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4929.508797] LustreError: Skipped 1 previous similar message [ 4930.406934] LustreError: 405797:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772309620 with bad export cookie 11555121765330361557 [ 4930.409731] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4930.410046] LustreError: 405797:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4936.571246] Lustre: server umount lustre-MDT0001 complete [ 4936.572869] Lustre: Skipped 1 previous similar message [ 4941.132390] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 4942.120159] Key type lgssc unregistered [ 4942.232400] LNet: 410714:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4942.235483] LNet: Removed LNI 192.168.204.147@tcp [ 4942.518657] Key type .llcrypt unregistered [ 4942.519724] Key type ._llcrypt unregistered [ 4953.659951] alg: No test for adler32 (adler32-zlib) [ 4954.412396] Key type ._llcrypt registered [ 4954.413448] Key type .llcrypt registered [ 4954.451147] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 4960.085661] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 4960.584961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4960.764627] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 4960.846114] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 4960.848593] LNet: Accept secure, port 988 [ 4962.431136] Key type lgssc registered [ 4962.762944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4965.501472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4967.435531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4969.292457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4971.163077] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4973.290357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4973.315567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4974.389683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4974.399180] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4974.431377] Lustre: lustre-MDT0000: new disk, initializing [ 4974.454109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4974.462761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4975.852386] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4979.824918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4979.844736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4979.876190] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4979.896487] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4979.897664] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4979.898075] Lustre: Skipped 1 previous similar message [ 4979.938379] Lustre: lustre-MDT0001: new disk, initializing [ 4979.969077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4979.981364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4979.986218] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4981.298177] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4984.964127] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4986.726669] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4988.789568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4988.813950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4988.816931] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4988.904458] Lustre: lustre-OST0000: new disk, initializing [ 4988.905973] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4988.910664] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 4988.938093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4990.213695] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 4993.821610] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4994.025840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4994.033443] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4995.209509] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4995.278760] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4996.554053] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4996.617286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4999.136933] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4999.137162] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4999.138804] LustreError: Skipped 1 previous similar message [ 4999.142127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4999.144255] Lustre: Skipped 1 previous similar message [ 5003.287491] Lustre: server umount lustre-OST0000 complete [ 5005.791707] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5005.794163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5005.798958] Lustre: Skipped 1 previous similar message [ 5005.800417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5009.376387] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5010.688909] Lustre: server umount lustre-MDT0000 complete [ 5011.924040] LustreError: 414794:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772309702 with bad export cookie 11368403892203914395 [ 5011.926536] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5011.927796] LustreError: 414794:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5012.045526] Lustre: server umount lustre-MDT0001 complete [ 5014.770270] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 15:15:05 (1772309705) [ 5017.797921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5020.541404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5020.704332] Lustre: lustre-MDT0000: new disk, initializing [ 5020.715749] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5020.735393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5022.210239] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5025.012485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5025.124458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5026.577633] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5029.008849] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5030.782569] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5035.488545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5035.489062] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5035.489222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5035.489227] Lustre: Skipped 2 previous similar messages [ 5035.492569] Lustre: Skipped 1 previous similar message [ 5037.505260] Lustre: server umount lustre-MDT0000 complete [ 5038.700865] LustreError: 419930:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772309728 with bad export cookie 11368403892203915249 [ 5038.702759] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5038.704321] LustreError: 419930:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5041.748706] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 15:15:32 (1772309732) [ 5043.917770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5044.073789] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5044.095470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5045.558542] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5048.495056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5050.122639] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5052.829989] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5054.744274] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5056.942697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5058.432629] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5060.769523] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5068.244627] Lustre: Permanently deactivating lustre-OST0000 [ 5068.250249] Lustre: Setting parameter lustre-OST0000-osc.osc.active in log lustre-client [ 5074.924367] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5075.271859] LustreError: 425637:0:(mgs_llog.c:1551:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002020 [ 5075.274850] LustreError: 425637:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 5075.276850] LustreError: 425637:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 5075.875030] LustreError: 425685:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 5075.877398] LustreError: 425685:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 5082.583655] Lustre: server umount lustre-OST0000 complete [ 5082.585221] Lustre: Skipped 1 previous similar message [ 5083.473469] LustreError: 425934:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 5083.476430] LustreError: 425934:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 5084.640910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5084.641110] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5084.641688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5084.641695] Lustre: Skipped 1 previous similar message [ 5084.645465] Lustre: Skipped 1 previous similar message [ 5090.271150] Lustre: 426031:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772309774/real 1772309774] req@00000000045bd9f6 x1858401363594496/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1772309780 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5091.589202] LustreError: 423331:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772309781 with bad export cookie 11368403892203915788 [ 5091.591583] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5091.595038] LustreError: 423331:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5096.927165] Lustre: 426233:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772309781/real 1772309781] req@00000000fac8c1e6 x1858401363595520/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1772309787 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5097.051507] Lustre: server umount lustre-MDT0001 complete [ 5097.052989] Lustre: Skipped 1 previous similar message [ 5099.836525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5101.350670] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5102.294955] LustreError: 427142:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -6 [ 5104.412623] Lustre: 427487:0:(mgs_llog.c:1311:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.147@tcp [ 5111.263111] Lustre: 427586:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772309795/real 1772309795] req@00000000a1729647 x1858401363595840/t0(0) o251->MGC192.168.204.147@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772309801 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5115.023580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5115.172995] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5115.178175] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5115.182357] Lustre: Skipped 1 previous similar message [ 5115.200816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5115.203505] Lustre: Skipped 2 previous similar messages [ 5116.641447] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5119.394161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5119.482314] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5120.955228] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5123.352092] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5125.263593] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5125.924524] Lustre: Permanently reactivating lustre-OST0000 [ 5125.929355] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [ 5125.931432] Lustre: Skipped 2 previous similar messages [ 5147.672486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5147.789305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5147.791705] Lustre: Skipped 1 previous similar message [ 5149.184982] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5151.619463] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5164.513057] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5164.513668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5164.516323] LustreError: Skipped 1 previous similar message [ 5164.520432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5164.523092] Lustre: Skipped 4 previous similar messages [ 5165.589448] Lustre: server umount lustre-OST0000 complete [ 5165.591410] Lustre: Skipped 1 previous similar message [ 5169.632964] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5169.639482] LustreError: Skipped 1 previous similar message [ 5170.200969] LustreError: 429895:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772309860 with bad export cookie 11368403892203916733 [ 5170.202935] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5170.204288] LustreError: 429895:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5174.862946] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 5175.840710] Key type lgssc unregistered [ 5175.955416] LNet: 432766:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5175.958384] LNet: Removed LNI 192.168.204.147@tcp [ 5176.265945] Key type .llcrypt unregistered [ 5176.267936] Key type ._llcrypt unregistered [ 5182.234972] alg: No test for adler32 (adler32-zlib) [ 5182.987391] Key type ._llcrypt registered [ 5182.988606] Key type .llcrypt registered [ 5183.021990] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 5188.604354] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 5188.933922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5189.151558] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 5189.223831] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 5189.225304] LNet: Accept secure, port 988 [ 5190.799103] Key type lgssc registered [ 5191.117445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5193.987055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5196.021793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5197.939154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5199.880164] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5202.053219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5202.076809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5203.160079] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5203.171206] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5203.211187] Lustre: lustre-MDT0000: new disk, initializing [ 5203.232537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5203.239308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5204.687268] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5208.697214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5208.723292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5208.759383] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5208.776362] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5208.778053] Lustre: Skipped 1 previous similar message [ 5208.812891] Lustre: lustre-MDT0001: new disk, initializing [ 5208.842797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5208.853945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5208.857896] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5210.176147] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5213.892553] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5215.753491] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5217.951461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5217.970699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5217.972990] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5218.058071] Lustre: lustre-OST0000: new disk, initializing [ 5218.060329] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5218.084176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5218.175943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5218.178763] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5219.458143] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5223.236423] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5224.667886] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5224.745045] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5226.098333] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5226.168503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5228.514055] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5228.514247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5228.523987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5228.525758] Lustre: Skipped 1 previous similar message [ 5233.633291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5233.634923] Lustre: Skipped 1 previous similar message [ 5238.753065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5238.754831] Lustre: Skipped 1 previous similar message [ 5241.311073] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5241.369572] Lustre: server umount lustre-OST0000 complete [ 5243.872307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5243.872898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5243.875614] Lustre: Skipped 2 previous similar messages [ 5248.704118] Lustre: server umount lustre-MDT0000 complete [ 5248.992889] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5248.998523] LustreError: Skipped 1 previous similar message [ 5249.927992] LustreError: 436028:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772309940 with bad export cookie 381586740559293412 [ 5249.930799] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5249.932772] LustreError: 436028:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5250.059619] Lustre: server umount lustre-MDT0001 complete [ 5252.787115] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5253.311648] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 15:19:04 (1772309944) [ 5255.478579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5255.629579] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5255.649779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5257.016440] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5259.707209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5259.821203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5261.155521] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5263.439535] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5265.194699] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5267.348932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5268.798248] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5271.173633] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5271.809825] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5273.259390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5280.735745] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5280.738009] LustreError: Skipped 1 previous similar message [ 5280.739189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5280.742903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5280.744565] Lustre: Skipped 1 previous similar message [ 5284.834852] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5284.837415] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5285.865294] Lustre: server umount lustre-OST0000 complete [ 5289.952886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5289.953838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5289.956394] Lustre: Skipped 1 previous similar message [ 5289.958065] Lustre: Skipped 2 previous similar messages [ 5293.254455] Lustre: server umount lustre-MDT0000 complete [ 5294.481858] LustreError: 440614:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772309984 with bad export cookie 381586740559294252 [ 5294.484737] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5294.484855] LustreError: 440614:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5299.115896] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 5300.039151] Key type lgssc unregistered [ 5300.148394] LNet: 444475:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5300.150891] LNet: Removed LNI 192.168.204.147@tcp [ 5300.430141] Key type .llcrypt unregistered [ 5300.431410] Key type ._llcrypt unregistered [ 5308.948831] alg: No test for adler32 (adler32-zlib) [ 5309.700476] Key type ._llcrypt registered [ 5309.702233] Key type .llcrypt registered [ 5309.737914] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 5309.822250] Key type .llcrypt unregistered [ 5309.823790] Key type ._llcrypt unregistered [ 5316.175871] alg: No test for adler32 (adler32-zlib) [ 5316.927287] Key type ._llcrypt registered [ 5316.928242] Key type .llcrypt registered [ 5316.962589] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 5317.309337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5317.486486] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 5317.557574] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 5317.559274] LNet: Accept secure, port 988 [ 5319.135127] Key type lgssc registered [ 5319.526531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5322.114211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5323.253944] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5323.267218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5324.675297] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5327.421043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5327.516140] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5327.519866] LustreError: Skipped 1 previous similar message [ 5327.522253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5328.545945] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5328.550672] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 5328.552155] LustreError: Skipped 1 previous similar message [ 5328.845496] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5331.190534] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5333.351426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5333.472903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5334.886303] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5343.715244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5343.720890] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5343.721209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5343.727716] Lustre: Skipped 1 previous similar message [ 5348.247757] Lustre: server umount lustre-OST0000 complete [ 5352.929097] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5352.932921] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5352.939039] Lustre: Skipped 1 previous similar message [ 5352.941720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5353.951976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5353.952479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5355.717109] Lustre: server umount lustre-MDT0000 complete [ 5356.949955] LustreError: 449263:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772310047 with bad export cookie 9806316137883017927 [ 5356.952076] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5356.955873] LustreError: 449263:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5357.077591] Lustre: server umount lustre-MDT0001 complete [ 5362.031270] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 5363.153881] Key type lgssc unregistered [ 5363.275320] LNet: 450065:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5363.278331] LNet: Removed LNI 192.168.204.147@tcp [ 5363.598431] Key type .llcrypt unregistered [ 5363.600112] Key type ._llcrypt unregistered [ 5372.510553] alg: No test for adler32 (adler32-zlib) [ 5373.262379] Key type ._llcrypt registered [ 5373.264300] Key type .llcrypt registered [ 5373.306787] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 5373.739725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5373.899343] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 5373.983620] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 5373.985841] LNet: Accept secure, port 988 [ 5375.567121] Key type lgssc registered [ 5375.896652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5378.555518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5395.039457] LustreError: 451498:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.147@tcp: failed processing log, type 1: rc = -5 [ 5426.783253] LustreError: 451498:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.147@tcp: failed processing log, type 4: rc = -110 [ 5456.479180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5457.916794] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5460.358359] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5462.571041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5479.116505] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5479.142954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5479.170437] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [ 5480.484615] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5483.202168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5483.321291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5484.746115] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5487.225475] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5489.101482] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5493.730191] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5493.730373] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5493.736507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5493.742894] Lustre: Skipped 1 previous similar message [ 5497.493528] Lustre: server umount lustre-OST0000 complete [ 5498.848232] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5498.848596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5498.849062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5498.855925] Lustre: Skipped 2 previous similar messages [ 5503.968025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5503.969660] Lustre: Skipped 2 previous similar messages [ 5504.839123] Lustre: server umount lustre-MDT0000 complete [ 5506.048987] LustreError: 451505:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772310196 with bad export cookie 7913382397568358482 [ 5506.051943] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5506.052665] LustreError: 451505:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5506.171651] Lustre: server umount lustre-MDT0001 complete [ 5510.565284] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 5511.496485] Key type lgssc unregistered [ 5511.613312] LNet: 455193:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5511.616202] LNet: Removed LNI 192.168.204.147@tcp [ 5511.888006] Key type .llcrypt unregistered [ 5511.889111] Key type ._llcrypt unregistered [ 5520.656949] alg: No test for adler32 (adler32-zlib) [ 5521.408262] Key type ._llcrypt registered [ 5521.409373] Key type .llcrypt registered [ 5521.441120] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 5521.875144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5522.029504] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 5522.106399] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 5522.108184] LNet: Accept secure, port 988 [ 5523.687090] Key type lgssc registered [ 5524.024352] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5526.572354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5527.712139] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5527.725865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5529.037762] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5531.729940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5531.842853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5533.145145] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5535.362596] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5537.114047] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5539.259262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5539.364862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5540.696963] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5543.194402] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5543.460702] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [ 5552.607803] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5552.611291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5552.615302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5553.632838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5553.633134] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5553.634567] Lustre: Skipped 1 previous similar message [ 5553.637855] Lustre: Skipped 1 previous similar message [ 5556.169796] Lustre: server umount lustre-MDT0000 complete [ 5556.906656] LustreError: 458605:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.47@tcp arrived at 1772310247 with bad export cookie 1808983894400289614 [ 5556.914495] LustreError: 458605:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5557.527108] LustreError: 458605:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772310247 with bad export cookie 1808983894400288998 [ 5557.531066] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5558.752068] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5558.752593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5558.752595] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5558.755667] LustreError: Skipped 1 previous similar message [ 5558.759580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5558.764119] Lustre: Skipped 1 previous similar message [ 5562.784365] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5562.784725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5562.792422] LustreError: Skipped 1 previous similar message [ 5563.671920] Lustre: server umount lustre-OST0000 complete [ 5565.880474] Lustre: server umount lustre-MDT0001 complete [ 5570.366483] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 5571.295530] Key type lgssc unregistered [ 5571.408359] LNet: 460457:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5571.410644] LNet: Removed LNI 192.168.204.147@tcp [ 5571.700685] Key type .llcrypt unregistered [ 5571.701845] Key type ._llcrypt unregistered [ 5580.235994] alg: No test for adler32 (adler32-zlib) [ 5580.987318] Key type ._llcrypt registered [ 5580.988486] Key type .llcrypt registered [ 5581.025469] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 5581.349029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5581.572853] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 5581.652149] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 5581.654194] LNet: Accept secure, port 988 [ 5583.231075] Key type lgssc registered [ 5583.539300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5585.951450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5587.089218] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5587.103571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5588.436397] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5591.150410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5591.273058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5592.658983] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5595.025472] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5596.810890] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5598.918307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5599.028880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5600.366333] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5602.084579] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:161 [ 5602.825607] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5607.393426] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5607.399070] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5607.401163] Lustre: Skipped 1 previous similar message [ 5609.292593] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.47@tcp (stopping) [ 5610.746744] Lustre: server umount lustre-MDT0001 complete [ 5612.512119] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5612.512354] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5612.512492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5612.512499] Lustre: Skipped 1 previous similar message [ 5612.513085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5617.632168] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5617.632538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5618.456297] Lustre: server umount lustre-OST0000 complete [ 5625.823149] Lustre: 464964:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772310309/real 1772310309] req@00000000108fdece x1858402014735040/t0(0) o251->MGC192.168.204.147@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772310315 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5625.894485] Lustre: server umount lustre-MDT0000 complete [ 5631.557406] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 5632.636583] Key type lgssc unregistered [ 5632.753340] LNet: 465718:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5632.757545] LNet: Removed LNI 192.168.204.147@tcp [ 5633.046585] Key type .llcrypt unregistered [ 5633.047588] Key type ._llcrypt unregistered [ 5641.993236] alg: No test for adler32 (adler32-zlib) [ 5642.744313] Key type ._llcrypt registered [ 5642.745233] Key type .llcrypt registered [ 5642.780792] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 5647.900113] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 5648.290937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5648.452123] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 5648.527749] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 5648.529383] LNet: Accept secure, port 988 [ 5650.111094] Key type lgssc registered [ 5650.480108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5653.436542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5655.412672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5657.312762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5659.271496] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5662.057281] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 5665.018075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5667.227727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5667.252600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5668.356916] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5668.367815] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5668.410158] Lustre: lustre-MDT0000: new disk, initializing [ 5668.436310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5668.442796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5669.857996] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5673.220396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5675.402826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5675.426144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5675.430243] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5675.518441] Lustre: lustre-OST0000: new disk, initializing [ 5675.520462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5675.522925] Lustre: Skipped 1 previous similar message [ 5675.541714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5676.932096] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5680.620754] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5683.723569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5685.868315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5685.892815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5685.930676] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5685.953668] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5685.968201] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5685.979775] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5686.007453] Lustre: lustre-MDT0001: new disk, initializing [ 5686.034647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 5686.039581] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 5686.059803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5687.515900] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5691.225569] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 5691.291971] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5692.618100] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 5692.684759] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5696.481972] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5696.481982] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5696.487043] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5696.491328] Lustre: Skipped 1 previous similar message [ 5700.300854] Lustre: server umount lustre-MDT0001 complete [ 5701.600893] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5701.604840] LustreError: Skipped 1 previous similar message [ 5706.722589] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5706.725722] LustreError: Skipped 1 previous similar message [ 5711.841842] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5711.845497] LustreError: Skipped 1 previous similar message [ 5712.863921] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5712.866694] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5712.870223] Lustre: Skipped 1 previous similar message [ 5712.872372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5712.874606] Lustre: Skipped 1 previous similar message [ 5716.959991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5716.960464] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5718.777473] Lustre: server umount lustre-OST0000 complete [ 5732.831162] Lustre: 474112:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772310416/real 1772310416] req@00000000e5af43e0 x1858402084994368/t0(0) o251->MGC192.168.204.147@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772310422 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5732.914995] Lustre: server umount lustre-MDT0000 complete [ 5738.833517] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 5739.808234] Key type lgssc unregistered [ 5739.932481] LNet: 474868:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5739.935428] LNet: Removed LNI 192.168.204.147@tcp [ 5740.232304] Key type .llcrypt unregistered [ 5740.233406] Key type ._llcrypt unregistered [ 5753.768881] alg: No test for adler32 (adler32-zlib) [ 5754.520404] Key type ._llcrypt registered [ 5754.521362] Key type .llcrypt registered [ 5754.558432] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 5754.653505] Key type .llcrypt unregistered [ 5754.654830] Key type ._llcrypt unregistered [ 5761.247890] alg: No test for adler32 (adler32-zlib) [ 5761.999317] Key type ._llcrypt registered [ 5762.000628] Key type .llcrypt registered [ 5762.043640] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 5762.388996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5762.575729] Lustre: Lustre: Build Version: 2.15.8_1_gea526c4 [ 5762.652102] LNet: Added LNI 192.168.204.147@tcp [8/256/0/180] [ 5762.653668] LNet: Accept secure, port 988 [ 5764.231123] Key type lgssc registered [ 5764.580976] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5767.465354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5769.931521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5772.016434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5774.273881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5774.310955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5775.404872] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5775.416308] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5775.459855] Lustre: lustre-MDT0000: new disk, initializing [ 5775.484348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5775.491802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5776.807142] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5780.702954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5780.727204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5780.763181] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5780.781123] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5780.783727] Lustre: Skipped 1 previous similar message [ 5780.823531] Lustre: lustre-MDT0001: new disk, initializing [ 5780.854205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5780.866252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5780.870587] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5782.180040] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5785.810415] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5787.706355] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5790.329925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5790.362330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5790.366352] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5790.470072] Lustre: lustre-OST0000: new disk, initializing [ 5790.473401] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5790.501616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5792.021367] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5795.811345] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5798.900482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5798.903798] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5804.000620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5804.002020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5804.003988] Lustre: Skipped 1 previous similar message [ 5804.007315] Lustre: Skipped 2 previous similar messages [ 5809.120479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5809.122045] Lustre: Skipped 2 previous similar messages [ 5809.474199] Lustre: server umount lustre-MDT0000 complete [ 5810.587899] LustreError: 479204:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772310500 with bad export cookie 1465037402491710975 [ 5810.590710] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5810.592768] LustreError: 479204:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5810.694502] Lustre: server umount lustre-MDT0001 complete [ 5817.311081] Lustre: 481608:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772310501/real 1772310501] req@00000000984e2348 x1858402204530624/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772310507 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5817.382338] Lustre: server umount lustre-OST0000 complete [ 5820.770604] Lustre: DEBUG MARKER: == conf-sanity test 73: failnode to update from mountdata properly ========================================================== 15:28:31 (1772310511) [ 5821.129436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5823.393161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5823.549912] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5823.571479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5824.882280] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5827.616588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5827.744853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5829.134321] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5831.486857] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5833.260160] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5835.311075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5835.334919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5835.396966] Lustre: Found index 0 for lustre-OST0000, updating log [ 5836.704271] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5838.504267] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5 to 0x0:33 [ 5839.023490] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5842.592733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5842.593423] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5842.598446] Lustre: Skipped 1 previous similar message [ 5846.229739] Lustre: server umount lustre-OST0000 complete [ 5848.032181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5848.032721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5848.035546] Lustre: Skipped 1 previous similar message [ 5848.036991] Lustre: Skipped 1 previous similar message [ 5848.415124] LustreError: 485293:0:(import.c:355:ptlrpc_invalidate_import()) lustre-OST0000_UUID: timeout waiting for callback (1 != 0) [ 5848.420139] LustreError: 485293:0:(import.c:378:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000be5eae31 x1858402204541376/t0(0) o8->lustre-OST0000-osc-MDT0000@1.2.3.4@tcp:28/4 lens 520/544 e 0 to 0 dl 1772310538 ref 2 fl UnregRPC:ENU/0/ffffffff rc -5/-1 job:'' [ 5848.429621] LustreError: 485293:0:(import.c:389:ptlrpc_invalidate_import()) lustre-OST0000_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting for them to error out. [ 5853.151949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5853.155292] Lustre: Skipped 2 previous similar messages [ 5856.925055] Lustre: server umount lustre-MDT0000 complete [ 5858.085072] LustreError: 482413:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772310548 with bad export cookie 1465037402491712298 [ 5858.087966] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5858.089176] LustreError: 482413:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5860.748639] Lustre: DEBUG MARKER: == conf-sanity test 75: The order of --index should be irrelevant ========================================================== 15:29:11 (1772310551) [ 5862.580319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5864.154366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5865.970971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5867.485498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5872.392171] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 5875.195423] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 5878.240423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5880.302591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5882.246074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5884.229950] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5886.379872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5886.399540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5886.478444] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5886.490137] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5886.532881] Lustre: lustre-MDT0000: new disk, initializing [ 5886.558095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5886.561476] Lustre: Skipped 1 previous similar message [ 5886.568233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5887.941668] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5891.993340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5892.017882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5892.049467] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5892.064537] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5892.066529] Lustre: Skipped 1 previous similar message [ 5892.108149] Lustre: lustre-MDT0001: new disk, initializing [ 5892.148320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5892.153499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5893.541793] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5897.243704] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5898.988330] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5901.222456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5901.244842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5901.247343] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5901.354538] Lustre: lustre-OST0000: new disk, initializing [ 5901.356641] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5902.584249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5902.587819] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5902.863539] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_default_debug -1 all 8 [ 5906.568966] Lustre: DEBUG MARKER: oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5908.023478] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5908.097492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5909.487073] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5909.557192] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5912.219703] Lustre: server umount lustre-OST0000 complete [ 5912.222754] Lustre: Skipped 1 previous similar message [ 5917.663680] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5917.665756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5917.669209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5917.670966] Lustre: Skipped 2 previous similar messages [ 5919.711083] Lustre: 493126:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772310603/real 1772310603] req@000000001cb270d3 x1858402204556672/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1772310609 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5920.224068] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5920.227845] LustreError: Skipped 1 previous similar message [ 5920.944885] LustreError: 489439:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772310611 with bad export cookie 1465037402491713341 [ 5920.947975] LustreError: 166-1: MGC192.168.204.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5920.948728] LustreError: 489439:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5928.380189] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing set_hostid [ 5931.082473] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing load_modules_local [ 5934.598409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5937.095276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5939.137669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5941.270525] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5941.801223] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5823 sec ========== 15:30:32 (1772310632) [ 5972.611168] Lustre: DEBUG MARKER: oleg447-server.virtnet: executing unload_modules_local [ 5973.507662] Key type lgssc unregistered [ 5973.615332] LNet: 498187:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5973.618475] LNet: Removed LNI 192.168.204.147@tcp [ 5973.871480] Key type .llcrypt unregistered [ 5973.872501] Key type ._llcrypt unregistered